Over a half dozen years or so, I have noticed a trend in our software engineering world towards collecting less and less data about what we do and how well we are doing it.  This would be contrary to lets say professional sports where data collection (i.e. stats) is a well know practice and pastime, plus on the rise, whereas in our professional software engineering world, it appears to be on the decline.  I wonder why?

A history lesson first.  A fantastic short paper called, “Lessons learned from 25 years of process improvement: The Rise and Fall of the NASA Software Engineering Laboratory” provides some real insight to a problem that now seems to be rippling through our industry en mass.  I would say the Software Engineering Laboratory (SEL) was ahead of its time and has seen some possible future outcomes that may be coming true for some (most?) organizations in the corporate world.

The SEL collected software engineering metrics on process improvement from 1976 to 1991 and beyond.  I am not going to summarize the lessons here, you can read the excellent paper for that, and if you were/are ever a part of a real software engineering organization, none of the lessons learned should come as a surprise.  But I think that is the real lesson to be learned from this paper and the amazing contributions to software engineering that came out of the SEL and that is, very few organization are actually practicing real software engineering process improvement.  And in fact, I would suggest that the latest trends in software development lifecycles, like Agile, XP, Scrum, et al are actually going the other way.

Allow me to explain where I am coming from; here is one of the lessons learned from the SEL: “Establishing a baseline of an organization’s products, processes, and goals is critical to any improvement program.”  In most other industries, this is a given, including professional sports.  Heck even in non professional sports, like your college track and field events, you establish your baseline for the 100 yard/meter sprint and constantly trying to improve on that as a simple example.  Yet, in the software engineering world, where is our baseline?  We have embarrassing baselines like spending millions and billions (note - how embarrasing, in my home country!) of software that is of poor quality or doesn’t even work.  Maybe that is why we don’t measure ourselves?

We have “new” software development lifecycle methodologies that promote no collecting metrics on how well we are developing software.  Its astounding and disturbing, not only to me but to businesses that are forking over millions of dollars for software development projects that seem to have nothing more than an ad hoc process of going wherever the project meanders, no matter how long or at what cost, let alone even validating if the right problem is even being solved in the first place.

How do I know this?  I have been in the software engineering business for 18 years and had the privilege to have experienced many a software shop ranging from multinationals like Eastman Kodak, Motorola to midsize companies and to small start ups, including my own, in several different industry verticals and locations in North America.  I had the good fortune of being taught Software Engineering Management, a 2 year post graduate program from the University of Calgary, taught by Motorola University, where each of the instructors had many a year experience and stories about software engineering projects, both failures and successes and most importantly, what the differences were/are.  I have kept in touch with past employees/friends in the industry to hear interesting anecdotes over a virtual beer.  And finally, I spend a lot of time researching and reading about software engineering in general. 

And my conclusion is that we as an industry are (way) off track and seem to be heading further off track with the invention and adoption of new methodologies that are not based on (any) actual factual experience/metrics/historical data like the 25 years of solid software engineering experience that the SEL has produced.

Why do I believe that?  I see it everyday and in several companies that I may come in contact with due to my job.  I don’t need any further proof than that.  Just even the simple metric of “establishing a baseline” of anything in your organization should be proof enough for you.  What data have you collected for x software engineering projects you have undertaken, in any of the companies you have worked for?  I bet it is a small percentage, if any at all.

Why is that?  I think the SEL folks hit it on the head with one of their lessons learned, “It is difficult to make an engineering organization aware of the importance of software engineering to their mission.” I would substitute engineering organization with “any” organization.  Software is everywhere and in everything.  It is almost as ubiquitous as the telephone or power, yet we have no way to measure its quality, effectiveness or how productive we are at designing and constructing it.  In fact, I would say that that any given orgranization is not even aware of how important the role of software plays in their very own organization.

And we software engineers, developers, programmers, coders, craftsman, or whatever we believe in (I believe that software design and construction is an engineering discipline) are our own worst enemies.  Behold, I give you programming.reddit.com where almost 100,000 subscribers argue over which is the best is programming language, ever, every day, year after year.  Or CodingHorror.com, with +100,000 subscribers, with its blog name, light hearted or not, sends the wrong message to the masses that coding results in a horror show. Or even Hacker News… How would any organization looking to solve a serious business problem that costs millions of dollars in software engineering take anyone of these seriously?  Or just assume that the whole lot of us is represented by these blogs and news feeds?  No wonder we have credibility problems.  No wonder we don't collect stats!

So what we do about it?  Just like anything, it takes a lot of hard work and some critical mass for an organization to realize that software may be a big part of their business and that maybe they should seriously consider looking at how software could be engineered for their business based on real software engineering principles and not on the latest and greatest fad of the month.  I don’t know about you, but if I was to spend a million dollars on any software engineering project, I would want to be sure that it was on budget, on time, had some level of quality in it and actually solved the right problem.  Establishing a baseline of what you are doing and how well you are doing it, is the start of real software engineering.