I came from the R&D electronics engineering world in the 80s and got into software development in 1991.  After 15 years in the software dev biz, the only conclusion I can draw is compared to the industrialized electronics world I came from, the software development world is decidedly unindustrialized.  In addition, we are going backwards instead of forwards in our progress towards industrialization.  I am genuinely concerned about the future of our software industry.

 

Nonsense you say?  Let me put some context and perspective around my statements.  In the 80s when I was still in the electronics engineering world, software computer programs (i.e. CAD/CAM) were on their way to the top of the Technology Adoption Curve,  particularly with the release of AutoCAD in 1982, which soon brought a level of design automation to the electronics engineering world that was literally revolutionary.  I lived it.  Here is a concrete example of what I experienced.  Instead of handcrafting printed circuit board layouts by hand, within a couple of years, we were operating several sophisticated computer (i.e. CAD) programs that could assist with the design of the layout of the board cheaper, better, and faster than I could do it manually.  What the CAD programs didnt have was my experience so in the hands of a skilled and experienced operator meant the ability to put cheaper goods, with much more quality in the hands of consumers, faster.  It also meant some serious electronics industrialization has happened over the last twenty-five years - almost feels exponential in my opinion.

 

I used the term unindustrialized in the first paragraph, what do I mean be that?  I mean that in our software development world, we do not have any (real!) predictable or repeatable processes for the specification (i.e. design) of software.  And please dont tell me about yet another great process methodology**.  Worse yet, when we do design and construct a software artifact, it invariably is the wrong thang.  We also cant scale in anyway to design large scale architectures.  Finally, our software development world today is based upon designing software from first principles, rarely do we leverage other peoples excellent work, including CAD systems and/or approaches that worked in other, now industrialized industries (e.g. electronics).  I see the latter being the biggest hurdle to overcome as it is a cultural change instead of a technical one.

 

** Note In 1995 I worked at Eastman Kodak in Rochester, NY.  Kodak Equipment Commercialization Process (KECP) was a new product introduction (NPI) process methodology that has evolved for decades we have nothing this evolved in our software world.  Further, some would say that in order to make a process predictable and repeatable, you need to be using the same tools over and over again or at least for a period of time to establish a baseline and then a feedback loop.  Our software tools keep changing every six months (or earlier), so we cant even establish a baseline let alone a feedback loop  Kodak knows a thing or two about industrialization and we in the software industry could learn from that.

 

What do I mean by going backwards?  I feel that we are going backwards in two areas, programming languages and more importantly, design environments that can produce design specifications, and no I do not mean CASE tools.  We are woefully lacking Computer Aided Design tools that are now best practices (and have been for years) in other industrialized industries like electronics, manufacturing, building construction to name a few.  Do you think it is unreasonable of me to expect better software design tools when I was handed exponentially more powerful tools in the electronics industry years earlier?  Thats my gripe, still hand crafting single lines of code after 15 years in the industry.  Something is wrong, man!  Maybe I am real naive or born in the wrong era, but you would think that we in the software industry, after 15 years (try 30!) could come up with better design tools than a glorified text editor, now mockingly called an Integrated Development Environment.  No?

 

So, programming languages, how can I say we are going backwards?  Preposterous!  Well, whatever happened to Smalltalk?  Yes, every programming language is a foreign language, some easier to learn than others and thats my point.  Have a quick read of this short article from 1991 to see what I mean.  From my industrialization perspective look at how incredibly simple and efficient Smalltalk is!  The language consists of 5 keywords (Java and C++ have +50) and the entire language specification can be expressed on a single sheet of paper!  Isnt this the pinnacle? The state of the art?  Its language roots are +30 years old and seem to be one of the most mature programming languages available.  What happened?  Not only was the language simple, but the development environment was equally as simple along with the most seamless source control I have ever worked with.  Again, what the heck happened?  Whats wrong with Smalltalk?  No, I dont mean strongly typed versus dynamic or VM or GC or anything technical.  I mean is it because our industry is so immature we cant see a good thing when we see it?  Is it because our industry is dominated by a handful of companies which dictates the software programming languages market?

 

The multiple programming languages I use today and the IDE I am in are far more complex by at least an order of magnitude.  Further, rather than optimizing a language that is simple to learn and effective, we proliferate choices for the sake of choice, nothing technical here.  I know of one vendor that supplies 7 different graphical user interface technologies, each complete with its own framework class library and even more IDEs.  7 UI technologies from one vendor?  How is this making it easier (for anyone)?  How do I know which one to choose?  How do I become expert in any of them, as I still need use my primary language (C#) and another incredibly large framework class library, .NET 2.0.  If anything, it is getting harder and harder to develop custom applications, not easier.

 

XAML is a brand spanking new declarative programming language from Microsoft.  This raises the level abstraction from an impetrative programming model (e.g. C#) to a declarative programming model.  So if I need a day job and live in the MS world, I would take a long hard look at XAML.  It goes far beyond then just another user interface mark-up language (using Windows Presentation Foundation).  Sure the view is separated from the model, yay, but that was a Smalltalk invention (i.e. MVC).  However, the crucial invention here is that XAML is a language that enables developers to specify a hierarchy of common language runtime (CLR) objects with a set of properties and logic.  Meaning specifying a hierarchy of any .NET objects using a declarative language.  Hmmm there is a CAD program for designing software in there somewhere.  Key point to get - just like Objective-C provided an interface to any objectified clib to be plugged together like electronic integrated circuits.  Remember Brad Coxs Software ICs?

 

When I was working at a software company called the AND Group in the early 90s, an extremely sharp programmer named Tim Wasko (look at you now Timbo!), showed me Brads invention on a NEXTSTEP computer.  I was blown away and so was Tim.  Brads Interface Designer and Integrated Circuit (IC) library approach to supplying and assembling software components using a common interface (Objective-C) rang true to me then as it does now, 15 years later.  It is an excellent analogy to the tools and approach we took in the electronics engineering world.  Not only did it work, but software programs like AutoCAD had a major impact on the entire product development process for the electronics industry (and others), meeting the requirements of industrialization.  Finally, a piece of software that can really claim automation of processes and offers scalability that no human could do.  AutoCAD (and similar) revolutionized the electronics engineering through the 80s.  Note that the tool enabled the process of industrialization and in fact, you hear little about the product development process, cause it is well known.  However, in our industry, because we have no standard way of designing software, all we have is process methodologies with dumb a$$ names.

 

Brads Objective-C System Building Environment is interesting as most people classify his invention as yet another programming language.  Unfortunate, as the crucial invention here was that the language (i.e. building environment) allowed systems to be easily designed based upon assembling composite objects using an Interface Designer and IC Library.

 

This approach is very similar to the electronics world where our designers are various CAD programs that would specify what we wanted in terms of electronic schematic circuit diagrams, circuit board layouts, mechanical drafting design, etc., which all represented the specifications (i.e. design) of what was to be implemented (built, constructed, made real, realized, whatever you want to call production).  We also had our IC libraries called electronic parts catalogs that we could order parts that met our specifications and then could be assembled onto our now made printed circuit board whose layout was based upon a circuit diagram specification, all of which we specified (i.e. designed) in some sort of CAD tool.

 

Why do I think this comparison to the CAD world of electronics and our own?  I believe this is the path to software industrialization.  I recently talked to an old mate (hi Darren!) I used to work with in the early 80s when we were part of an electronics R&D shop.  He went on to get his Masters in Electrical Engineering and has been working in Nortels R&D department as a RF Designer for the last 8 years.  I asked him what some of the advances were in the way of specification (i.e. design) tools.  He laughed and said that they have a directory tree full of CAD tools used to specify what they want built.  Most if not all of the tools are visual design tools with a few (visual) simulation tools.  In other words, they are domain specialized tools, with their own languages for each aspect of the specification that needs to be designed.  In fact, if you were to classify it to our software world, one could say that each one of these CAD tools encapsulated their own Domain Specific Language (DSL).

 

I would also say that Brads System Building environment, from 15 years ago, was also a DSL that objectified clibs using a standard interface (i.e. pluggable palettes and a common design surface).  In other words, the visual design raised the level of abstraction to a point where you were assembling prebuilt C libraries that plugged into a standard interface.  That meant any supplier of clibs could objectify their library and have it plug into the design environment.  One analogy is like Visio where you buy a set of 3rd party symbols (like electronic symbols for example) that plug into Visio as a standard palette in which now you can interact with on the drawing surfaces and design circuits.

 

In fact, this is the point of the analogy.  In the electronics world, the electronics circuit schematic diagram with its symbols and lines represent a design specification.  Every other artifact and/or transformation is derived from the design specification.  Note that everyone uses the same (domain) specific language to design electronic circuits.  This language consists of numerous symbols.  Each symbol is an abstraction of the real thing.  There are symbols for resistors, capacitors, transistors, integrated circuits, etc.  But note that the symbols are simply interfaces.  Each interface has a definition, like a resistor symbol typically has two connectors and between those connectors is where the specification of the resistor is specified, in which there are a million combinational values. Or another way to look at it is someone elses spec sheet is the resistor value.  In fact for every symbol or interface on the design surface is where you indeed plug in a spec sheet that came from a catalog or that you have to custom build yourself or a third-party builds it for you.

 

Of course, in the electronics world, we use CAD programs to draw our schematic diagrams using these standard symbols.  We still dont have these types of specifications tools in our software design world.  In fact, we still dont get what software design is.  Have a look at this 1992 article on, What is Software Design.  Software design is still the 1000s of lines of code - in other words, the actual program listing is the only real design specification.  This program listing was hand-crafted one source code line at a time.  The closest activity I can think of that is similar to writing software is like writing a novel, but in a foreign language next blog topic.

 

So what does this all mean?  It means that we in the software industry have built all sorts of CAD, CAM, etc., tools for virtually every industry but our own.  How ironic.  In my opinion we are still using hammers and chisels to sculpt one-off master pieces.  How many e-commerce applications in the world have been hand-crafted? Millions of them. Functionally, they are all virtually identical and still today we keep hand crafting them, one source code line at a time.  This is what really makes me mad about our industry, the all so mighty not invented here syndrome or must build from first principles or whatever it is to avoid using someone elses perfectly fine programming language, development environment, etc.  So every year we just re-invent the same thing over again and give it a new acronym.  If I have to learn one more programming language or one more class library I am going to quit this nonsense and grow oranges, or since I am in BC, grow some BC b$d.

 

Until we as the software development industry get serious about producing some real CAD tools for designing software, we are going to hand-craft our profession into extinction.  I can tell you that the future of software industrialization wont look like and that is what it is today building the software world one source code line at a time.

I have a great deal of respect for Brad Cox.  In fact, consider this essay a tribute to Brad. Ten years before he wrote, Planning the Software Industrial Revolution, I was living in the integrated circuit world of R&D electronics, where it was already an industrialized industry.  I could order chip, card and rack level components from catalogs and plug them together though a set of standardized interfaces that solved a set of problems that I had specified in an electronic schematic diagram.  During that time, I was also a participant in industrializing printed circuit board layouts that initially took 6 to 8 weeks of hard manual labor on a light board to, ironically, using a computer program to read an electronic schematic diagram (i.e. specification) and producing the printed circuit layout in a matter of minutes.  In other words, I experienced first hand what industrialization was and meant in the electronics industry.

 

When I first got into the software industry in 1990, I thought it was going to be similar to the industrialized electronics industry I was used to.  I could order software components (like integrated circuits) from catalogs and using a schematic diagram, CAD like software program of some sort, assemble those components to solve a set of problems.  Nothing could be further from the truth.  Then I read Brad Coxs article and realized what I took for granted in the electronics engineering world may not even happen in my lifetime in the software world.  It was quite a shock to me.  I had no idea the software world was so far behind, and in fact, completely unindustrialized.

 

Flash forward 15 years and I have performed just about every role one could imagine in the software industry, including running my own software company.  I feel like I have gone nowhere in this industry and in fact, in some respects gone backwards.  I still write the same source level code I did 15 years ago and build most everything from first principles.  Sure technologies have changed, but we (meaning everyone in this industry) are still building our software world, one source code line at a time.  As Brad says, grains of sand where bricks are needed.  Whats wrong with our software industry?  Why dont we learn from the successes of other industrialized industries?  Are we that arrogant that we have to do everything from first principles over and over again?

 

In Brads article he discusses the importance of Blanchards pattern lathe and that the real crucial discovery made was, that implementation tools were insufficient unless supplanted by specifications tools capable of determining whether parts complied to specification within tolerance.  As Brad points out, this discovery has not yet occurred in software.  In my opinion, 15 years later, it still remains undiscovered.  Making software tangible and observable, rather than intangible and speculative, is the first step to making software engineering and computer science a reality.  Brads sentence still applies today.

 

Ironically, computers and computer programs have helped other industries become fully industrialized.  In 1982 AutoCAD came to the world and revolutionized the industrial design and manufacturing industries, including my printed circuit board story above.  The craft of drafting has not gone away, but the tools used have raised the level of abstraction to the point today where those specifications are now items that I can order out of a catalog.  For example, if I want to design a house, I dont have to invent the drafting symbols that represent my house, they already exist and in fact, can be ordered from a catalog.  Further, I can order the entire house specification (funny enough called an architectural blueprint from a catalog and customize it (using standard symbols) to what I want.  That specification is then used to implement (i.e. build) the house.  Some people would say in the software world we have that through UML. When I showed a business person a UML use case diagram, he said, Stickmen?  Thats all you have is stickmen? And then he proceeded to laugh uproariously and just walked away.  At the time I felt angry and thought another bozo that does not get it.  But as years went by, I realized he was totally right.  How embarrassing for our software world where the best we can do on the specification side is stickmen.

 

So we need better specification tools, thats for sure what else do we need?  As Brad writes, the programmer shortage can be solved as the telephone-operator shortage was solved by making every computer user a programmer.  I wholeheartedly agree.  This would dramatically increase the chances of industrializing our software world.  At the very least it would help us with the most frustrating part of our world and that is human machine interface design.  Generally speaking, software engineers and engineers simply dont get it, for whatever reason.  Lets look at several examples of what I mean and tell me I am wrong.

 

Take your average DVD player. Did you ever happen to notice that the hardware (DVD player) does not control the software (DVD disc)?  One of my commenters (thanks Michael K) on my blog said, I want a DVD player that works like a VCR.  When I press fast forward on a VCR - the recording goes forward. With a DVD - it is up to the disk whether I can fast forward it or not - total lunacy!  Its my player it should do what I tell it, now!  And just how many DVD players are there in the world?

 

How about automated phone attendants? A sure way to increase the blood pressure of even the calmest West Coaster, particularly the new and improved attendants that respond to your voice commands instead of key commands. Insert your favorite voice command here   The fact companies purport that they are customer driven by using automated attendants is a joke or is it that they dont get it? (ha!).  When I speak to a company, I want to hear a human being answer the phone and one that knows the inner workings of the company to provide me with a layer of abstraction between the companys goofy organizational structure/behavior and myself.  S/he is the public interface to the company that hides the inner workings from me.  The thing about it is that we used to have it, so what happened?

 

What about ATMs?  Well, it is a matter of convenience.  The fact that I can quickly get money, usually without waiting in line, is what I want from a users perspective.  So that part of the abstraction is good, but the human machine interface could do with another tweak.  That is, if I have only one account, then dont give me account choices.  I get asked for checking, savings, and other, every time I use the machine, no matter where I go in the world and I have only one account.  How hard can it be?

 

Even the firmware in my wifes German precision engineered washing machine is koo koo.  It has lots of intelligence built-in, but the machine does not automatically shut-off by itself.  It will beep beep, beep at you until you come and manually turn it off.  Going through the 27 page manual, I found a way to at least turn-off the beeping.  However, months later it came back and I forgot the magic key combination to turn it off and I refuse the read the frickin manual again.  How can this be?  How many millions of washing machines have rolled off the production line in the last 25 years?  And still cant get down to the one button interface called wash clothes, and dont bother me again.

 

Or how about, Windows Vista, 5 million lines of code, 5 years later and Hasta La Vista Baby!  As you will see they have also opted for aesthetics over functionality.  What does the customer really want? An extremely fast, low memory, transparent to the user and completely reliable operating system.  Thats it!   Again, how hard can it be?  Yet, wait till you see it and by the way, you will need to buy a new computer to run it.  That from the largest and most successful software development company in the world today.

 

Finally, while not really software based, is an excellent example of poor human to machine interface design.  The building that I work in is having its washrooms renovated and you guessed it, upgraded to automated faucets, soap dispensers, urinals and toilets.  Yes, the toilet is fully automatic, with no manual flush.  Of course, the renovations are occurring floor by floor meaning that for the people on the floor that is being renovated will need to go up or down a floor, which means those washrooms are rather busy.  So what if the automatic toilet does not flush?  And there are people waiting?  I can tell you that one of the toilets, (3rd floor, right stall), is batting a .500 average.

 

The first lesson for anyone becoming a software programmer is to become a designer first.  There is an excellent book called. "The Design of Everyday Things", written around the same time as Brads essay.  This book should be required reading for every designer, software, hardware, industrial, art or otherwise.  One story discusses the design of doors and the visual clues it presents to the user to determine does this door push forward or draw backward and if I need to turn a knob or push on a bar or what exactly to make it work.  Yes, something as simple as a door, we still cant get right.  Aesthetics win over functionality, almost every time cause thats what the consumer perceives s/he wants.  Me?  I just want the door to open like in old school Star Trek, complete with the cool sound.  Oh yeah, you will recognize the book as it has a picture of a red tea pot on the cover with the spout and the handle on the same side.  Nice design!

 

So what does this have to do with industrializing software?  Everything actually.  Until consumers demand better software, cheaper and faster, we will continue to handcraft solutions one source code line at a time that cant be measured for conformity to any users specification.  Eventually consumers will revolt or go elsewhere where someone else has figured out a better way to do it, just like when consumers got fed up with American cars, they went to Japan.

 

I am somewhat encouraged by the fact that Domain Specific Languages (DSLs) are making a resurgence as I believe this is the right path towards industrializing software development and meets Brad Coxs vision.  DSLs raise the level of abstraction to a point where a visual model specification is used to specify the implementation and can verify its correctness.  AutoCAD can be classified as a DSL as it allows non-draftsperson or non-engineer to still produce a drawing that is accurate enough for someone to implement it.  For example, from a catalog, I can order a drawing of a piston, put it in a CAD program and modify a dimension, save the electronic file output, take it to a milling shop and get my pistons manufactured, and yet I know nothing about the industry.  Now thats industrialization.

 

A Challenge to My Software Developer Peers

 

I challenge my software developer peers to help industrialize our software development industry.  Brad Cox has made a major contribution not only with the article he has written, which I reference here, but also with Objective-C System Building Environment that he invented.  My very small contribution to software industrialization is co-inventor of a DSL called Bridgewerx that allows a Business Analyst (not a programmer) to specify (AutoCAD like) an application integration specification that is then automatically implemented, without any programming knowledge required.  And before you crucify me for plugging a product I no longer have any financial interest in, I did it because I am personally sick and tired of writing the same source level code over and over again for the last 15 years.  I want higher level abstractions in better tools.  What are you doing to raise the level of abstraction in our software industry?

 

I leave you with the last paragraph that Brad Cox wrote 15 years ago in his article.  It is just as applicable today as it was then and probably still applicable when I am no longer around.

 

I only wish that I were as confident that the changes will come quickly or that we, the current software development community, will be the ones who make it happen.  Or will we stay busy at our terminals, filing away at software like gunsmiths at iron bars, and leave it to our consumers to find a solution that leaves us sitting there?

There is an excellent article written in 1994 called, Softwares Chronic Crisis whose conclusion is, Despite 50 years of progress, the software industry remains years-perhaps decades-short of the mature engineering discipline needed to meet the demands of an information-age society.  That same year, while I was taking Software Engineering classes taught by Karl from Motorola University, I was told the same thing.  Motorola shared their own software metrics to validate what the industry was saying.

 

Again, that same year, (what was it about 1994?), a CHAOS report came out that said the US spends $250 billion per year on software development for roughly 175,000 projects ranging from $140K to $2.3 million per project.  Out of those projects:

Flash forward 10 years later and the Standish Group has this update about our software development track record:

So what does this mean?  It means we as the software development industry (still) have major problems in developing software on time and budget.  Shhh, dont tell our customers.  If you read the reports, there are a variety of reasons why we have this record, and not all of it on to ourselves.  However, after 15 years in the software development biz, including running my own software company, I can safely say that these metrics are quite real.  In my experience, the only way to deliver on time and budget is to give up features.  Thats the state of the art today.  Even the largest and most successful software company in the world has the same problem on making software development a predictable and repeatable process, Vista 2007, Fire the Leadership Now!"

 

Softwares Chronic Crisis is right on the money back in 1994 and still is today in my opinion.  The root cause is that our software development industry has yet to be industrialized.  This is in comparison to other industrialized industries like electronics, mechanics, etc.  These industries have matured from one-off custom development to mass customization (i.e. industrialization).  Mass customization has the following characteristics:

Our software world currently does not yet have these characteristics of industrialization.  Our software development products are designed and constructed one source code line at a time.

 

So what does this have to do with IT is a commodity?  Some people think IT is a commodity.  While I think the hardware side of IT, including some basic firmware applications, maybe a commodity, the software side (bought s/w products or custom dev) is still far, far away from being a commodity.  Look at our track record.  Look at your own products/projects.  How many bugs (do you know about) are in it?  According to a recent NIST Study on the impact of software bugs, conducted in 2002, "Software bugs, or errors, are so prevalent and so detrimental that they cost the U.S. economy an estimated $59.5 billion annually, or about 0.6 percent of the gross domestic product. More than half of the costs are borne by software users, and the remainder by software developers/vendors."

 

If you could visibly see the thousands of bugs in most commercial software products, would you still buy it?  We do it all the time.  I like the fact that companies sell "packaged" enterprise application software and then suggest to you that you pay for services to install and configure the software and it will cost less than a custom build.  Maybe, maybe not.  I personally know of several configurations of ERP systems in the Oil and Gas industry in Western Canada that have cost 10 to +100 times the cost of the product itself.  The product cost is $1 million.  If I were a Customer of such a service, I would also want my own personal jet (and full-time pilot) for that kinda of money. Cmon!

 

How is IT (i.e. software development) a commodity?  Software development is still an incredibly skilled and massively labor intensive process.  Our development tools lags platform technology.  What do I mean by that?  While we have platforms that provide the technology to do (most) everything we want, our development tools are so low-level that we are (still!) handcrafting every solution as a custom one-off, one source code line at a time.

 

How do we industrialize our software development industry?  Next post we will look at how other industries have done it, and ironically, using computer technology as an enabler to industrialize their own business domain.

Some years ago I had the opportunity to participate in a software technology bake-off. Technology selections or bake-offs go something like this, Dear Vendor, you have been selected to show your wares/skills at our esteemed company.  No, you dont get paid for this privilege, but you have one week to complete the tasks in this envelope that you will perform on-site.  And if you win, we are not sure what we are going to do anyway. So, dance vendor, dance!

 

OK, while I am being a bit tongue in cheek, it was exactly like this.  I view bake-offs as the easy way out for CXOs that dont know what they are doing or dont want to do their homework themselves.  In my experiences, it has been the former.  Unfortunately, for the vendor, it is a very risky proposition to participate if the org does not understand what they are doing or getting involved in.  At the time of this particular bake-off, it was all about Enterprise Application Integration (EAI) even though Service Oriented Architecture (SOA) principles applied, this buzzword had yet been invented.  In other words, this org was a very early adopter of SOA and did not know it.  This meant that Geoffrey Moores Technology Adoption Curve is in full play here.

 

While I am going to name the technologies that were used in the bake-off, I do want to make clear that years later, I came to the conclusion that it did not really matter what technology was used.  The reality is that any of the middleware products would have worked for this org.  As an analogy, they were looking for a Cadillac solution and at the time thought BizTalk was an econobox and all they really needed was a bicycle.  It has been my experience that this is the case for most orgs from a technology selection perspective.

 

Back to the story, but first a little background.  The orgs CIO and his crew were hardcore Unix fans and therefore were horrified to find a Microsoft product (BizTalk) that had penetrated their domain.  Our pro services shop was called in to perform a complicated workflow across a few applications related to one of their core business processes for a particular department which got us in under the radar.  When we presented our prototype (live from our servers) in front of 30 some people, everyone seemed to be quite pleased.  At that moment, the CIO stood up and said, dont too excited, as we have to go through a formal selection of which middleware technology our esteemed org will use.  Just so you know, this was the beginning of the end for the CIO who was let go sometime later.

 

Yours truly got grilled by the CIO and Unix crew on more than one occasion as to why BizTalk.  I was so naive.  I dutifully and enthusiastically explained the features and benefits, blah, blah blah.  It reminds me so much of the commercial that has a cardboard cutout of a salesman that said, So, how much software would you like to buy?  As a technical person playing a sales roll, I cant help feeling that way.  However, I happened to be President of this pro services company and just bit the bullet all for the company right?  Behind the scenes, the CIO flew in a top sales person from Vitria (their Unix fav) who sold a bunch of licenses to the org., even before the bake-off began.  The other two players in the EAI selection were Tibco and WebMethods.

 

The funniest part of the bake-off, depending on who you ask, is when we opened the envelopes, we quickly realized that the third-party consulting company hired to develop the EAI tasks, did not know what EAI meant.  The majority of the tasks were what was traditionally known as a data pump.  That is to pump data from one database to another, sometimes indirectly through staging tables and/or file shares.  The third-party company that did the spec did not know that the A in EAI stands for application not database integration.

 

Depending on who you speak to, maybe there was nothing funny at all.  Take my business partner for example.  As it turned out, I happened to be on vacation in the northern regions of Canada at the time (hey, its the North Pole and got daily phone calls on how choked my partner was.  Complicated further by the fact we were not getting as much local Microsoft support that we thought we were and since our business model was using independent contractors, we could not engage our regular crew cause this was not paying gig.  Consequently, my partner had to pull double shifts for that week.  The only other time I saw him this mad was when one of our business partners tried a hostile takeover behind our backs within the first three months of incorporation, but that is another story.

 

At the end of selection process, with lots of internal lobbying going on, it was a tie!  I have never heard of such a thing in my 15 years in the biz. Totally hilar.  And the way it was rationalized was that all Unix based projects would use Vitria and all MS based projects would use BizTalk (the company was already a 50/50 split between Unix and MS applications).  Part of the inside joke here is that the whole point of a middleware product was to be programming language and platform agnostic and could integrate any disparate applications, which was lost on the CIO that called for the bake-off in the first place.  Further, the independent third party consulting company that put together the bake-off tasks also happened to perform Vitria pro services and actually offered up their services.  Isnt that a conflict of interest? 

 

As it turns out, Vitria stayed on the shelf and over a dozen BizTalk projects were designed and constructed for the org with our pro services shop getting most of the work.  A personal thank you to Bobby Keen!

 

I learned later that the org spent close to $2 million in the bake-off over a year time frame.  I could not believe it, but after someone on the inside explained to me what all had happened and how many people were involved, I could see how it happened.  The org could have saved themselves this money or have it put directly to actually solving business problems.  So why dont they get it?

 

There was no joy of software on this one.  Ultimately, none of us got it cause the orgs entire IT shop got dismantled as a large outsourcing org swooped down and took the business away from all of us.  However, as mentioned earlier, Geoffrey Moores Technology Adoption Curve was in full effect here and I believe we all fell while Crossing the Chasm.  Subject of my next post.

I have been employed in the software development biz for 15 years.  I used to really like what I did.  Now I am not so sure I like what I see in our industry.  I see vendors plying their wares with cheesy marketing campaigns like Microsofts Dinosaur ads  or Oracles Fusion middleware that is hot pluggable (what marketecture).  I am not sure why others arent disappointed by this or maybe I am so nave that this is just the way it works in our industry.

 

When I first started, I was the eager programmer stayed up all hours to ply my craft.  It was fun and I have no regrets.  I was fortunate enough to move from C, which I really did not get, to Smalltalk, which I did get.  Since then, I have had opportunity to use many programming languages and frameworks.  15 years later, I see them as all the same still low level tools, not much beyond assembly language in some respects.  We still laboriously toil over cryptic lines of a foreign language to get it understood by the machine (i.e. compiled) and in the end some human being is going to interact with the code you wrote and hopefully it fulfills whatever function it was supposed to do.

 

Amidst this is the whole world of quality, metrics, process improvement, Agile, CMMI, ISO, Scrum, Extreme Programming, etc.  To me, its all the same, most of it just common sense.  It still boils down to functional decomposition from Knuths Truths written many years ago.  Make the problem small enough for the humans to understand and have reasonable expectations on planning a solution for the problem.  I used to work for a division in Kodak (CMM Level 2) and division in Motorola, where we were a CMM Level 5 shop.  My disillusionment of the whole process improvement world was born at these companies, who were not really interested in quality from a product perspective, but more from a marketing perspective, so they could get a sticker that says they have a quality process.  Bureaucracy was what it really was.

 

Over 15 years ago, I came from the electronics engineering world where most of what we did in that industry was fairly predictable and repeatable.  I thought when I joined the software world, it was more or less the same.  After 5 years in the industry I came to the conclusion that the software world is not even close to being predictable or repeatable.  In 1994, I read the Standish companies CHAOS Report, which confirmed to me that our software development track record is absolutely abysmal.  The latest report says we have improved, but still our record states that for any given project it is still a crap shoot if the project succeeds or not.  As a closet perfectionist, this does not sit well with me.

 

After I got out of the quality game, I dived deeper into programming and buried myself with projects for a number of years.  Some were successful and others failed miserably.  None of them failed for any technical reason or staff skills, even though staff variability in productivity can vary 20 to 1 according to industry gurus.  My experience meter suggests programmer variability is much higher than that.  Programmer productivity aside, I was still seeing the truth in the CHAOS report, every project was a crap shoot.  How to increase the odds for success?

 

I know, I will start my own company and I can control everything, muh, ha, ha, ha. So I did, I formed my own company called 5by5Software with another like minded programming buddy and for the first couple of years, we were successful in turning a profit in our little pro services world.  As I have written elsewhere, I am extremely fortunate to have worked with several brilliant programmers.  It amazes me to have people that can listen to a conversation and go away and within a short period of time, there is the solution, and with a little refactoring, solid as a rock.  Yet, with other people, where the specs have been written in infinite detail, only to be looked at with glassy eyes.

 

In our company, we did so many projects of a particular type that we saw a repeatable pattern and a business opportunity to become a products company, in which we did, called Bridgewerx.  After 4 years being President, I resigned cause I was dissapointed by the direction the company was taking from a business perspective.  Of course when you require investment, other people get involved and the original vision gets diluted along with your shares.  I am proud of what we invented because it does raise the level of abstraction of a particular problem domain.  That part feels good.  However, now what?

 

I could go to work for Microsoft, yet when I read Mini-Microsofts blog, I am amazed at the problems this huge company is having.  What about Google?  JavaScript you say, hmmm where is the challenge in that?  Maybe I will start up another company.  For anyone that has started their own software venture knows the effort required to be successful, it aint for the faint of heart.

 

Yeah, start another company thats it.  My objective is to build tools that help raise the level of abstraction in our industry, which in my opinion, we are sorely missing.  Where are the tools for a Business Analyst to model or draw business processes?  And I aint talking Visio.  Domain Specific Languages (DSL) look like a good candidate to start building tools from.  You can read an excellent DSL article by industry guru Martin Fowler.

 

DSLs to me are what is exciting and perhaps the next generation of programming tools for our industry.  I am not talking about a new programming language or a framework, I am talking about a fundamentally different way to design and construct software.  Rather than build solutions, build tools that build the solutions in an automated way.  This is what I mean by software industrialization. 

 

It is also the way of other industrialized industries that have matured from one-off, hand-crafted solutions to more of a product line engineering approach.  "I don't think the craftsman will be replaced, but the tools he uses will be provided by companies who provide (or are) software factories."  This quote comes from Steve Cook's blog and I happen to agree with it.  Just like the drafting table was replaced by a CAD machine doesn't mean one loses his or her's craft, right?

Many moons ago I worked with two brilliant programmers named Steve Langley and Barry Varga.  Steve was our boss at the time with 25 years programming experience and Barry became my partner in a software adventure called 5by5, now Bridgewerx.  One interesting note was that Barry and I became Steves boss when he worked for our company a few years later.

 

We worked for a SCADA company who built RTUs for monitoring electrical substations and sending telemetry to a SCADA application.  An RTU is a firmware device with analog to digital interface cards to measure substation line voltage, current, load, transformer oil coolant temperature, etc., and feeds that information to a SCADA system.  A typical SCADA system usually monitored many substations that collectively forms part of the grid.  A substation may need dozens of RTUs and literally hundreds of different firmware applications running to monitor an electrical grid.  We had a catalog of over 300 firmware apps a customer could choose from.  The company also built custom firmware apps for our customers.

 

Each firmware app had hundreds of configurable properties, with no default values or data masks or min and mix values, etc.  Multiply that by 20 RTUs per substation with 20 apps running, each with at least 100 have to be configured properties means that a bunch of poor guys had to hand enter 40,000 property configuration values in order to make the substation automation work.  A software program was developed to allow configuration of these properties and then upload the config file to the RTUs firmware.  It would take six months to commission a substation based on this approach.  The bitch and stitch from the customers and company was that this was way too long and costly.  Our job was to develop a 2nd gen configurator application that would provide automation for most, if not all, of these properties and therefore reduce the commissioning time from 6 months to 6 weeks.  That was our mission Mr. Phelps.

 

This first generation configurator was a classic example of 2-tier architecture and over-engineering.  Using Delphi, the UI was a table editor, meaning that grid controls exposed the database shema directly to end users.  Tsk Tsk.  However, at the time, considered state of the art.  End users either could think like tables (a whole generation of table thinkers and tinkerers was born during the 90s) or could not think that way and wanted a higher level abstraction.  The over-engineering is that the engineering department decided to make every possible firmware parameter, for every firmware app, configurable, never thinking about size or complexity or the future of substation automation - the very business they are in.

 

Enter our team whose task it was to build a replacement for the old configurator to make the configuration process simple and repeatable.  We were object and component guys then.  So when we did the UML on the problem set, it turned out to be incredibly hierarchical in nature.  An electrical grid contains 1 to many substations, a substation contains 1 to many RTUs, each RTU contains 1 to many firmware applications, each firmware application contains 1 to many configurable properties, and each configurable property had 1 to many attributes.  This is a gross oversimplification, but nonetheless it was just a set of hierarchies.

 

Our approach was to build a drawing tool that allowed the Configurator to drag n drop different RTU types from a toolbox onto a canvas and connect the RTUs to a network.  If you double click on the RTU, a physical picture of the firmware collection was displayed with the name of each application.  Double clicking on an app brought up a configuration wizard that had your typical express and advanced modes.  Express mode gave you some simple options of a known working configuration that you can default to so that the system can be up and running ASAP.  Advanced mode exposed all of the properties, plus their default values that have already been set.

 

It was VB6 days for the UI and our only real option was to use J++ for the business objects layer, being an MS shop plus a real object hierarchy that modeled the real world.  Steve gets code automation.  He developed an Excel spreadsheet that contained all of the attributes for each property for every configurable item for every firmware app.  So if we had a property, we could give it a name, description, default value, data type, min or max values, whether it was required or not plus a bunch of other attributes, but I think you get the idea.  We used Excel to capture the human readable semantics and syntax (a DSL) configuration language for RTU firmware applications.  Steve then wrote a VB6 program that read the Excel spreadsheet and code generated J++ abstract and impl classes and next thing you know we have all of our business objects done.  Every time the spreadsheet would change (which initially was often, then settled out), we would regen the abstract classes, but not the impl classes unless we chose to do so.  Of course the impl classes is were our specific implementation code was.  Now of course we have partial classes in the .NET 2.0 framework which makes it even easier for DSL applications.

 

I wrote the configuration wizards and Barry did a masterful job on the drawing tool in which we used a third party drawing component. So far, so good.

 

However, the Delphi Lead Programmer did not want to switch to MS.  Battle ensues.  I wish I could say we won, but we did not. Our product never made it into production, even though there is nothing wrong with the software and meets the business requirements of both the customers and company by reducing the time and cost to commission a substation with their products.  So what happened?  Just soap opera stuff because we did not have a strong executive team to tell the Delphi programmer that a business decision has already been made and we are phasing (y)our configurator out cause it is too costly to us and our clients.  Instead the Delphi programmer quit and then held the company hostage to hire him back on contract (at a much higher rate) to keep working on the Delphi Configurator.  Oh man, sideways.

 

The Joy of Software was how the three of us worked together building software that resulted in a cool product that also met the business requirements.  Our test customer, from a real power substation in the US, was thrilled.  The why they dont get it goes to (weak) executive management for simply not following through on a business decision that they had already made. 

So Why dont they get it?

Wow, you can really feel the frustration in the comments at Minis blog on the delay of Vista.   There were over 300 comments the last I looked.  And the comments are shocking, asking for the resignation of leaders, why the delay is good and bad, comparing software complexity between Adobe Photoshop and Vista, that the product was 5 years in the making and 5 million lines of code later and she is going to blow, whats better in the OS than XP2, devs vs PMs vs Testers, etc.  There is a wildfire at the largest and most successful software company in the world today.

 

Quite frankly I am a bit shocked, not at the wildfire, but by the comments.  To me, it is no wonder that Vista has been delayed.  Why?  Its obvious, software development is a non predictable and non repeatable process.  Thats it, thats the reality of our business.  In my mind, that is the root cause of why software is not delivered on time and budget.  All the rest of it is noise.

 

Despite all of the process methodologies we have today, we still cant determine when a software product is going to ship.  We in the industry continue to underestimate the size and complexity of virtually any size software product.  The business side of the industry continues to promise software products that are all singing and dancing to customers.

 

I have a lot of respect for people like Watts S. Humphrey for writing books that apply the most disciplined approach to software engineering that I have ever come across, and unfortunately, I read too many text books, mostly about programming.  Yawn.  Whats my point?  Even with this much rigor, we still cant accurately predict size of product, release date, total cost, etc.  What does that have to do with Minis blog on the delay of Vista?  Everything!  Despite any type of leadership issues, variability in skill sets, stack ranking, sizing and estimating techniques, etc., Vista was delayed for the simple reason that the act of software development is inherently not predictable or repeatable.

 

There is empirical evidence ad nauseum that supports the fact that software development is non-repeatable process.  We have evidence of thousands of projects not only being late, but failed miserably.  I would never have believed it myself when I first came to this industry 15 years ago, because my background was in electronics engineering, where generally speaking, we did have a predictable and repeatable process.  In fact, the design and construction of most physical objects is pretty much a predictable and repeatable process today.  Unfortunately, software does not abide by the laws of physics.

 

Mini, in an earlier post asked what was the value of Vista that he could explain to his Mom.  Thats easy, navigation of the file system has been greatly improved.  That is a core and key feature of any OS, how well can the user navigate the file system.  As explained in an earlier post, I thought MS had done an excellent job here.  The usability is indeed better, I have experienced it myself.  Because of the shortcut approach in the address bar of Vista Explorer, I can navigate around the file system with fewer mouse clicks.  What more can my Mom ask for?

 

Another question Mini asked along the same lines for Office 12 or 2007 or whatever it is going to be called.  Have a look at the state of the art in features in Word 2003.  It has 32 toolbars.  How could this happen?  This is a word processor.  What should a Word processor do?  Allow me to write any type of document in the easiest possible way.  Fortunately, Word 12 is much better in that regard with its context switching ribbons.

 

Microsoft should be focusing on reducing the complexity of their products not only from an end user standpoint, but also from a developers point of view.  Seven user interface technologies from one company? Whats wrong with this?  Rather then making it simpler, this is going the other way in complexity.

 

What I like about Office 12 plus severs like SharePoint version 3 is that there are developer tools that can customize a SharePoint site quickly and easily.  Sure it is FrontPage rebranded and upgraded to become SharePoint Designer, but the point is that a Systems Analyst can now modify SharePoint sites easily and even describe fairly complicated workflows, all without writing a single line of code.  What does that mean?  It means we can deliver customized solutions to our Customers quicker and not have to have a PHD in programming, meaning the tool appeals to a wider audience.  Thats the ticket MS, build more Power User tools that allow more people to manipulate your products using less code.  The same approach could be applied to the tools you are using to build Vista.

 

This is called raising the level of abstraction.  From where I sit as a 15 years software developer, this is what will help industrialize our software industry so that software development becomes more predictable and repeatable, and then the headline could be that Vista ships on time as predicted.  Till then, what a mess our industry is in, all indicative of Minis blog detailing the inner workings of the worlds largest and most successful software company in the world.

On occasion I am fortunate enough to participate in various Vendors early adopter programs in getting bits of products before they are publicly available.  WIFM is by the time the product goes mainstream, I will have had a number of pilot projects under my belt and be ahead of the technology adoption curve, which is what keeps me employed.

 

Of course the Vendors want something in return, usually a customer reference as the idea is to have a partner work with a customer to use the early software bits as a pilot project.  The good news is that you and the customer get to see a real early and generally working view of the product.  The bad news in being a bloody beta tester is that more often than not you jump through (sometimes massive) hoops, cause the product simply aint ready yet.  Thats the trade off.  However, Vendors could do more for their Beta testers as we will see.

 

The amount of time it takes to install these products and get them configured right, particularly when various dependant products are also in various Beta stages, makes it a necessity to use Virtual PC (VPC).  Virtual PC allows you to run a software emulated computer in your host OS.  So I am running XP as my host OS and the VPC image is Windows 2003 Server configured as an App Server, along with SQL Server 2005, VS2005, WinFX 3.0 Beta 2, Community Technical Previews (CTP) of various other bits and pieces.  The list goes on and on.  Something like a dozen items over two days to get it installed and it has to be done in a certain order.  Dear Mr. Vendor, would it be so hard to have all of the software on one DVD and a single button that says install this scenario, just like you have in SQL Server and VS 2005.  How hard can it be?  Btw, the VPC lifespan is about one month as new Betas and CTPs will appear, making your image obsoleto as it may (read: will) not be upgradeable as we shall see.

 

VPC works great, but you spend a lot of time loading ISO images and restarting the (virtual) computer.  Of course the VPC software emulation (even with 2 gigs RAM) is much slower than your host OS, but the advantage is that you can take snapshots of your VPC image after each item is installed and if things go wrong, you can copy your previously working image and be up again very quickly.  One problem is that our image is now 16 gigs in size and takes a while to copy across the network.  The other major advantage is that all of the developers are working on the same VPC image, which if your code does not work on someone elses image, its your code thats broke, not the config, which makes our requirement for single unit under test condition true.

 

Now enter the Beta product I have been testing. We have been working on Beta 1 for a little while and it took a while to get the image running right.  There also is little documentation.  Which is a bit unusual from this particular vendor cause generally they are very good on docs and code samples.  Dear Vendor, please at least have how the tools work documented so we can at least try them out and not guess how they work.

 

Then comes along a Beta 1 refresh.   This is the Vendors response from the crush of feedback for Beta 1, coming to the realization that we cant wait for Beta 2, so lets give em a refresh, which really means, whats the latest known good build, ok, ship it!  The documentation says that there is no upgrade path.  No upgrade path to this Beta 1 version nor will there be an upgrade path for future versions.  Now I know this is Beta software, but I think it is a bit inexcusable when the end user (the developers in this case, including me) need to go through massive hoops of uninstalling the previous version, including other add-on bits like WinFX, etc., and then installing the newer version, only to be told that you still have not removed enough of the old version to continue, even though according to Add and Remove Programs it looks like I have already uninstalled everything.  Argh!

 

So now what?  Back to a previous VPC snapshot, that now needs Windows updates, a restart , plus a newer version of WinFX with a restart, plus I needed a virtual DVD software emulator to mount the ISO images, in the virtual PC, (wha?) and man, on it goes and I have not even got to the new Refresh install!  Which by the way the refresh has a list of 20 components!

 

During the installation process on a restart, I got a message that said, Since Windows was first activated on this computer, the hardware on the computer has changed significantly (authors note no it has not).  Due to these changes, Windows must be reactivated in 3 days.  Do you wish to reactivate now?  I said yes, first to find out on-line that the number of licenses has been exceeded.  Then I was given a number to call which an automated attendant asked me to read a sequence of 40 characters (took forever) and then was told that this was not right and that I need to talk to a customer service representative (ooooohh a human being, how rare).  The customer support rep, who was very nice and gave me another 40 characters to enter in (after I read my characters back to him) and activate my version of Windows.  Btw, this was done before I uninstalled all of the previous Beta 1 software and tried installing the new products Refresh only to be told I had not uninstalled everything previously.  So I just trashed the entire image.  So much for the activation.  Double aarrgghh!!

 

So now I am installing everything under another VPC that I had saved off that does not have any of the original products Beta 1 bits on it for sure or the rest of the software I need for that matter.  I hope I dont get that reactivation message again.  Dear Vendor, if there is no upgrade process, please provide an uninstaller tool that can completely removes the Beta or CTP product thoroughly with absolutely nothing lingering around.  Would have saved me days of effort.  Triple aaarrrggghhh!!!

 

So what about the Refresh install?  It is almost growing season here in BC, maybe I will get a job picking grapes because it will be much easier and far less bloody than this!