It is also science, math, engineering, physics, , etc.  At least one other has a similar opinion.

So what then is the definition of Software Engineering?

Wikipedia’s definition, to me, is the most pragmatic:

"Software engineering is a profession and field of study dedicated to designing, implementing, and modifying software so that it is of higher quality, more affordable, maintainable, and faster to build.”

But what does that mean?  If you believe the above definition, then you believe that in order to achieve the CBF (cheaper, better, faster) factor, you need to measure, not only the results of your effort, but the effort itself.  Meaning your process - how good is it?  How predictable and repeatable is it?  Can it be better?

Our industry is so... young, that we still have not learned the simple lessons found in other engineering disciplines, like “measure twice, cut once.”  It so reminds me of that scene in Monty Python’s, the Holy Grail, where the Black Knight, “although supremely skilled in swordplay, he suffers from unchecked overconfidence and a staunch refusal ever to give up.”  That’s us in the software industry – refusing to believe that software engineering is just another profession, like electrical, electronics, civil, mechanical, chemical, industrial, and other engineering disciplines are.

It seems to me that at some point in time (hopefully soon), critical mass will drive software engineering into a fully fledged profession.  For those that follow, it has been underway for many years and following the path of other engineering disciplines.  Or at least, since my first career was in electronics engineering many years ago, I am witnessing a similar process path to what I was involved in the early 80’s in the electronics engineering world.

Since it is possible to become a licensed Professional Software Engineer in British Columbia, I have been going down that path for the last year.  It’s a multi-year process to become a licensed professional.  Why?  The requirements for becoming a licensed Professional Software Engineer are extensive.  And so are the domain specific areas of (PDF Warning!) the Software Engineering Checklist and Software Engineering Experience Guidelines. Also, BC lacks many professional software engineers, relative to other engineering disciplines, so getting my application reviewed by the number of people required by the process takes quite a bit of time.

What’s my point?  Software Engineering as a Profession will happen – it is just a matter of time.  Why am I so concerned about this?  To me it is all about the quality of what we as software engineers produce.  Several running jokes should explain what I mean.  A 30 year vet software professional once told me that if we managed money the way we managed our source code, we would all be in jail.  Or how about, if this was a car, it would only have three wheels, brakes that don’t work, and a gas pedal that gets stuck from time to time (oh the irony Toyota! and I own a Tundra – love it).  If you physically saw this car, would you buy it?  Probably not as it is obvious that it cannot even be driven off the lot.

Oh boy, but do we ever buy software like that?  Hell yes folks, the truth is that the software we buy, relative to the quality of other designed and engineered products like a Ducati 1198 (look at that page man – hard core engineering specs for a work of art!), is poor.

So am I worried about not having “licensed” software engineers?  No, that will sort itself over time as our discipline becomes a real bona fide, first class profession, like a lawyer, doctor or... engineer. It will simply become a requirement just as it is for Professional Engineers today in other disciplines.

What I am concerned about is our consumers accepting low quality software because they can’t “see” if the software has three wheels or four wheels.  The latter, of course, being what the spec asked for.  Eventually, as more and more software is used in our daily lives, people will insist on better quality. Well, that comes at a price, and some will want or demand that level of quality.  That’s a shop I would work for.  I would rather work on designing the next generation Ducati as opposed to the next generation Ford truck (no offense Ford, but you might want to put some art back into those highly CAD trucks).  And just a small aside – how come the car industry has not figured out that I would pay extra to get a retro car that “looked” exactly like a 69 Cuda or 70 GTO 0r 69 Camaro or 71 Trans-Am or... pick your retro work of art.  The body and interior is the replica, but the tech is the latest stuff.  Instead, they produce the Challenger that kinda, sorta looks sort like the original, but not really – so I don't want it.  I could also speak to Product Line Engineering here as the car industry has it down pat and only a small percentage of us in the software world even know what it means and even less having experience applying the technique in our domain.

So where is the art in Software Engineering?  All you talk about is discipline, measurement, control systems, etc. 

Right you are, but if you have been paying attention, you will have noticed I have already said what the art is.  The software is the art.  The process of creating the software is the engineering discipline.  And like other engineering disciplines, the passion drives the art.  Have another look at that Ducati, (or whatever does it for you) and tell me ain’t that the truth!

"Software reliability engineering relies heavily on a disciplined software engineering process to anticipate and design against unintended consequences."

Wikipedia Software Reliability

The six paragraphs in this explanation on Software Reliability are bang on in my experience. In fact, the statement above best describes the difference between project success and failure. The more disciplined the software engineering process the better quality the output, in all measurable aspects, is my conclusion after 18 years as a software professional. Btw, disciplined != bureaucratic || inefficient.

I wonder if software reliability engineering had been applied to the following web sites, it would have been less likely for me to see these messages:

The irony here is that these two messagess occurred on two seperate dates, not just 20 minutes apart

An Error Has Occured?  Pics or it didn't happen! Oh... well how about directing me to another download site that is known to be working?  I can't imagine what's in the "On-line Self Help."

No way man!  

Aeroplan, a fairly large transactional site, was down for two days. I wonder if software reliability was on anyone’s minds when the site was designed…

You may have noticed that some of the screen shots show maintenance messages, not error messages (and other reliability messages). I find it hard to believe that there is (still) the practice of scheduling "planned outages" for web application/site maintenance on the main public domain web site. First business rule in the world wild web, you are always open for business.

Sure, some of these sites have staggering amounts of traffic and processing. However, it is 2009, where computer hardware has never been cheaper and the most powerful at the same time. Content Management Systems and Ecommerce systems, have never been more plentiful, more mature, and the most featured for the least amount of dollars.

That’s one of the points behind having a disciplined "software engineering process."  One of the most important activities, in my experience, is designing for software reliability. No, I don’t mean life and death reliability, but given the budget constraints, what is the best or most acceptable "business risk" the web application/site is able to mitigate from a reliability point of view.

What do I mean by business risk in this context? Given that Aeroplan was not able to transact business for two days, let alone how it looked to "customers", I would be surprised if the executives signed off on that their main public web site could be down for a two day period.  And no, I don’t mean "high availability" or five nines uptime, albeit related, I mean software reliability. Even if you paid for five nines up time, it is very likely your software will error out (or you planned a maintenance window!) well before you hit that uptime number.  I would say that this is predetermined by how your software was designed and engineered.

I wonder when the "tipping point" will occur where the general public will stop accepting software that is the analog equivalent of missing a wheel on a car. Sure you can drive it, but not very well. That’s the ultimate problem with software, it is completely invisible to the majority of people, but if they saw it, and it looked like a car, then they would see that missing wheel, hear the engine misfiring, and see/smell the smoke bellowing out where canola oil was used instead of engine oil. The car is clearly a lemon and would not sell. Even if it did sell, there is a lemon law.

Yet software contunues to be built, and software products exist today, that are the equivalent of the lemon of a car, but there is no lemon law or even a guarantee of any sort for software. There is a reason for that. Some would say that software is too complex or at least no-one, other than NASA can afford the level of software reliability engineering that we comparatively see in our cars today. Do you think that is true?

Based on my experience, I don’t think so. In fact, over the mid term and long run of a software lifecycle, spending the engineering time to design the software correctly (or as best as possible given budget constraints) easily costs (considerably) less than just trying to bang the software out using the latest du jour software process methodology and then fixing the heck out of it for years to come. At least my own anecdotal experience supports that position.

These software quality issues were screen snapped over a couple week time frame while I was browsing the net.  What does it mean?  I feel it means a couple of things.  One is that I am one person, the smallest possible sample size, yet look at the poor quality software I encountered during a short time frame.  I can't imagine what the impact is on millions of users sampling (hundreds of ?) thousands of web applications/sites daily.  The other is there seems to be far and few between companies that actually practice designing and engineering software in a disciplined manner...

Updated Next Day - Ha! 

Gmail outage deprives millions of e-mail

Quick, tell me what is common between, “” and Software Engineering?   Answer: They are both stories about engineering and one has tragedy in human lives.

The Tragedy at Second Narrows is an absolutely compelling story about engineering design. The tragedy is the number of lives lost due to what appears to be an engineering design oversight.  Ironworkers Memorial is of very special meaning to me as I live near Vancouver and occasionally have the privilege to drive over the Ironworkers Memorial Bridge.  I have a lot of respect for the Ironworkers, Engineers and the entire craftsman who worked on this piece of art.  The art is that traffic flows across the bridge at the same speed as the #1 Highway in Canada – the user experience is seamless and that’s its beauty, it is also by design (i.e. it has been engineered that way).

The Ironworkers Memorial Bridge also has a special meaning to me as it relates to software engineering.  I believe software design and development is an engineering activity.  In fact, the more years I do this (since 1977 in electronics engineering and slowly moving to software engineering since 1991) the more I believe it.  So do others, including educational institutions. You can get a 4 year degree in Software Engineering  and the States have been doing it since 1996 when I saw it at RIT while working at Eastman Kodak in Rochester, NY.  Damn man, you can even get a PhD in Software Engineering.

Software Engineering in the province of BC is recognized as a professional practice and since 1999, you can apply to APEGBC to become a licensed Professional Software Engineer.  Aside from having an ABET recognized 4 year engineering degree, 4 years documented experience, 4 P. Eng.’s to stand up and vouch for your sorry ass, attend a 2 day law and ethics seminar or consume the equivalent in CD-ROM, and pass a 2 hour Professional Practice examine, there are additional requirements.  You need to fill out a Software Engineering Checklist and you need to complete the Software Engineering Experience Guidelines. 

It is very interesting to read over the last two documents, Software Engineering Checklist and Software Engineering Experience Guidelines.  The point being, if you want or need a description of what Software Engineering is and how to document it, spend a bit of time reading these short documents.  While a bit old, the first principles hold true, which is what I refer to the lost art in my title (more later). 

As mentioned previously, Software Engineering is a recognized professional practice in British Columbia.  Check out the benefits of becoming a licensed Professional Software Engineer. Meet the first licensed Software Engineer, Philippe Krutchen, who was licensed in 1999 in BC.

Software Engineering to many people have many different meanings.  Some think/feel software development or programming is an artistic endeavor.  Others think of it as craftsmanship. From my perspective, I agree with everyone. In fact, it is all those things and much more. Some people feel that the La Sagrada Famillia is art.  Some would say the craftsmanship that went into it was second to none.  But it was also engineered (i.e. designed). Check this out, Gaudi designed the weight-bearing of his buildings, upside-down with string and bearings.  So when you looked in the mirror, of course it was right side up.  I saw this first hand.  Engineering at its finest, well before modern engineering with all our fancy calculators and our StarTrek Communicators. 

So Software Engineering is design, art, craftsmanship and any other word that people want to use that is not engineering.  Call it what you will, I agree with you.  But it is also part engineering.  That to me is the lost art, specifically the engineering design part.

In fact, I would say Software Engineering, at its very core, has four fundamental activities: requirements analysis, design, code, and test.  Of course you have to deploy and operate, maintain, and a whole pile of other things (reports, UI enhancements, skinning, ad infinitum), but I just want examine these four core activities.  Let’s put some definition around what I am getting at:

• Requirements Analysis:  I don’t care if you use story points, scenarios, use cases, flowcharts, or animated puppets to gather and package software requirements.  The question is how accurate is your transformation and how good you are at repeating it.  Understand what I mean?  How do we as a team communicate “what” needs to be built?  To answer the question, I am suggesting use whatever methods and tools you want as long as the result (i.e. the transformation) is accurate and you can repeat that accuracy on multiple projects consistently and eventually, predictable.  That to me is the lost art of software engineering in requirements analysis.  What is the percentage accuracy of our requirements gathering and packaging? i.e. relative to the intent of what the users actually require.  And btw, if the users or community can’t tell you, in gory detail, what they want, then no amount of software is going to solve the problem.  Capiche?  What’s our percentage repeatability and predictability?  That’s the theory and in practice it does works.  But it is a lot of very hard work, so you really don’t see it too often in our field, hence the lost art.

• Design: Ah, the crux of software engineering.  Your choice of design methods and tools should be determined on which methods and tools will result in the least complex, but most (easily) extensible design possible.  In short, at its essence is a mathematical formula for reducing complexity, but the art is your application of the formula at the problem at hand.  Again how accurate is your design transformation relative to the requirements you used to produce that design?  Can you do it consistently as new and old problems come your way?  Whether you choose a formal design method like MIT’s Alloy, (if you have never used Alloy before, try it on for size, it really demonstrates software complexity through modeling. ) or a traditional OOD method, is really not the ultimate issue here – choose whatever method/tool yields the most accurate Requirements -> Design transformation.

• Code – it seems everyone knows how to code in their favorite programming language.  Again ultimately, from a software engineering perspective, use whatever programming language yields the least complex representation or transformation of the design.  Oh, yeah, I know what about the human factor – where are we going to find programmers?  Or the million other factors that occur when selecting a programming language. Does it really matter?  Not if you want to produce the least complex implementation.

•  Test – There seems to be a dazillion and one testing methodologies/tools. Does it really matter which one you use or what tool?  I don’t think so.  Again, what percentage accuracy is our testing against the 3 previous transformations? Oh sure, at what level of effort you ask? What was the requirement for testing accuracy and coverage?  Don’t have one, better get one.  More lost art.

Oh, I know I am greatly simplifying the above, but isn't that a lost art also?  It seems to me that year after year we further complicate our art, not simplify it.  One lost art of software engineering is trying for the most accurate transformations from requirements to test (and eventually deployment, operations, and maintenance.).

Another lost art it seems is designing for a least complexity solution or at least making it a priority.  But that takes time, commitment and data.  And that’s my point.  It appears the trend I see in our industry, and if Jeff Atwood’s blog on Software Engineering: Dead  is any indicator, reading the comments are an eye opening experience about people’s thoughts on Software Engineering.  I have been there myself, on more than one occasion, and in fact, is normal for most companies that are trying to design software, but really don’t know how.

The only way to improve what we do as Software Engineers is to measure what we do and how we do it.  Oh sure, that tired old saw.  I have seen it work with my own eyes to know the truth and value behind it.  Yet, we as an industry seem to have completely forgotten the value of measuring what we do and how we do it.  Other industrialized industries would cease to function without this feedback control loop.  Can you imagine pro sports without stats? It would never happen, but the trend in our industry seems to be moving away from measuring anything and just going for it.  I wonder why that is?

My hope is that over time Software Engineering, while a Professional Practice in British Columbia, becomes more widespread, and what I used to take for granted as standard operating procedure (SOP) in Electronics Engineering, will become SOP in Software Engineering.  But like anything, this is going to take time and education, particularly for folks that don’t know about the engineering part, or for others like me that have lost it from time to time.

 As an 18 software development professional, I have been waiting for this day for a long time.  Honestly, I thought this day would never happen in my lifetime, but it did and in a form I would have never guessed:

“The future of games development has been called into question after the EU Commission suggested developers provide a two year guarantee.” 

Two year guarantee on… games?  I would have thought that medical software or aviation/guidance software or any other software application/system that could put people in harms way would have been the first industries to have instituted guarantees on defective software. 

Never did it cross my mind that it would be the gaming industry, but then again, after thinking about it, it makes perfect sense.  Video games are a multi-billion dollar business targeted at mass consumers, a.k.a. the public.  So when lots of people’s games don’t work, something has to give.

The key clause is, "two year guarantee on all games in the event that a bug or glitch is encountered preventing you from completing the game and/or ruining the experience."

As the articles states:

“At present, retailers are not obliged to give a refund on a video game that has a bug or glitch that prevents a user completing a game. If the proposals become law, this could change as users would have the right "to get a product that works with fair commercial conditions".

Note it says proposal, but I say great, it is about time!  Why is it only in the software industry, people and corporations spend trillions of dollars on software and none of it is guaranteed?  No, I don’t mean delivery guarantees, or warranty periods, I mean outright quality guarantees. 

For me, as someone who develops software, I could not be happier.  This marks the day that there is enough critical mass in the public to demand better quality software products.  Really it means that software will need more engineering, as in software engineering.

“Software engineering is the application of a systematic, disciplined, quantifiable approach to the development, operation, and maintenance of software, and the study of these approaches; that is, the application of engineering to software.”

At age 50, I have had two careers, the first in electronics engineering, where the application of systematic, disciplined, quantifiable approach… was never even questioned, talked about, it was simply expected, standard operating procedure, institutionalized, etc.  But when I entered my software career in 1991, I was very surprised to find that in the software world, there were debates going on, still today, whether software development is engineering or art or science or…

Fortunately today, many universities are offering undergraduate degrees in software engineering and have defined the difference between computer science and software engineering. 

This means that younger generations will be more exposed to software engineering rather than us “old guys” where there was no such thing called software engineering when we went to university.

I also believe that software engineers should be licensed as Professional Engineers (P.Eng. in Canada).  In fact, in British Columbia, Alberta and Ontario, you can be licensed as a professional software engineer.  

You can read an excellent paper on, "Licensing Software Engineers" by Philippe Kruchten  From the article, “The only purpose of licensing software engineers is to protect the public.”  I totally agree.  Now Philippe suggests that licensing software engineers who designs  and develops software that can kill is protecting the public.  I agree with that as well.  But I would take bit of a broader definition that not only includes software that kills, but again protects the public from financial impact, which basically covers most business software, like ecommerce, banking, trading insurance and any applications that deal with money, but also even t the point of gaming software, which is what this article began with and the proposed two year warranty.

Perhaps a little reflection is required to let what I am saying fully sink in.  For any software development project that has any impact on the public, financial or otherwise, the software engineer responsible for “sealing” the blueprints should be a licensed software engineer.  A licensed professional software engineer is bound by professional practice and code of ethics.  That means that a licensed professional software engineer is legally responsible for the design of the software.  Think about that.  Aside from the responsibility, the way software is designed and engineered is going to change in our world.  I can only be grateful for that and like I said early on, it is about time.  Today is a red letter day!

Over a half dozen years or so, I have noticed a trend in our software engineering world towards collecting less and less data about what we do and how well we are doing it.  This would be contrary to lets say professional sports where data collection (i.e. stats) is a well know practice and pastime, plus on the rise, whereas in our professional software engineering world, it appears to be on the decline.  I wonder why?

A history lesson first.  A fantastic short paper called, “Lessons learned from 25 years of process improvement: The Rise and Fall of the NASA Software Engineering Laboratory” provides some real insight to a problem that now seems to be rippling through our industry en mass.  I would say the Software Engineering Laboratory (SEL) was ahead of its time and has seen some possible future outcomes that may be coming true for some (most?) organizations in the corporate world.

The SEL collected software engineering metrics on process improvement from 1976 to 1991 and beyond.  I am not going to summarize the lessons here, you can read the excellent paper for that, and if you were/are ever a part of a real software engineering organization, none of the lessons learned should come as a surprise.  But I think that is the real lesson to be learned from this paper and the amazing contributions to software engineering that came out of the SEL and that is, very few organization are actually practicing real software engineering process improvement.  And in fact, I would suggest that the latest trends in software development lifecycles, like Agile, XP, Scrum, et al are actually going the other way.

Allow me to explain where I am coming from; here is one of the lessons learned from the SEL: “Establishing a baseline of an organization’s products, processes, and goals is critical to any improvement program.”  In most other industries, this is a given, including professional sports.  Heck even in non professional sports, like your college track and field events, you establish your baseline for the 100 yard/meter sprint and constantly trying to improve on that as a simple example.  Yet, in the software engineering world, where is our baseline?  We have embarrassing baselines like spending millions and billions (note - how embarrasing, in my home country!) of software that is of poor quality or doesn’t even work.  Maybe that is why we don’t measure ourselves?

We have “new” software development lifecycle methodologies that promote no collecting metrics on how well we are developing software.  Its astounding and disturbing, not only to me but to businesses that are forking over millions of dollars for software development projects that seem to have nothing more than an ad hoc process of going wherever the project meanders, no matter how long or at what cost, let alone even validating if the right problem is even being solved in the first place.

How do I know this?  I have been in the software engineering business for 18 years and had the privilege to have experienced many a software shop ranging from multinationals like Eastman Kodak, Motorola to midsize companies and to small start ups, including my own, in several different industry verticals and locations in North America.  I had the good fortune of being taught Software Engineering Management, a 2 year post graduate program from the University of Calgary, taught by Motorola University, where each of the instructors had many a year experience and stories about software engineering projects, both failures and successes and most importantly, what the differences were/are.  I have kept in touch with past employees/friends in the industry to hear interesting anecdotes over a virtual beer.  And finally, I spend a lot of time researching and reading about software engineering in general. 

And my conclusion is that we as an industry are (way) off track and seem to be heading further off track with the invention and adoption of new methodologies that are not based on (any) actual factual experience/metrics/historical data like the 25 years of solid software engineering experience that the SEL has produced.

Why do I believe that?  I see it everyday and in several companies that I may come in contact with due to my job.  I don’t need any further proof than that.  Just even the simple metric of “establishing a baseline” of anything in your organization should be proof enough for you.  What data have you collected for x software engineering projects you have undertaken, in any of the companies you have worked for?  I bet it is a small percentage, if any at all.

Why is that?  I think the SEL folks hit it on the head with one of their lessons learned, “It is difficult to make an engineering organization aware of the importance of software engineering to their mission.” I would substitute engineering organization with “any” organization.  Software is everywhere and in everything.  It is almost as ubiquitous as the telephone or power, yet we have no way to measure its quality, effectiveness or how productive we are at designing and constructing it.  In fact, I would say that that any given orgranization is not even aware of how important the role of software plays in their very own organization.

And we software engineers, developers, programmers, coders, craftsman, or whatever we believe in (I believe that software design and construction is an engineering discipline) are our own worst enemies.  Behold, I give you programming.reddit.com where almost 100,000 subscribers argue over which is the best is programming language, ever, every day, year after year.  Or CodingHorror.com, with +100,000 subscribers, with its blog name, light hearted or not, sends the wrong message to the masses that coding results in a horror show. Or even Hacker News… How would any organization looking to solve a serious business problem that costs millions of dollars in software engineering take anyone of these seriously?  Or just assume that the whole lot of us is represented by these blogs and news feeds?  No wonder we have credibility problems.  No wonder we don't collect stats!

So what we do about it?  Just like anything, it takes a lot of hard work and some critical mass for an organization to realize that software may be a big part of their business and that maybe they should seriously consider looking at how software could be engineered for their business based on real software engineering principles and not on the latest and greatest fad of the month.  I don’t know about you, but if I was to spend a million dollars on any software engineering project, I would want to be sure that it was on budget, on time, had some level of quality in it and actually solved the right problem.  Establishing a baseline of what you are doing and how well you are doing it, is the start of real software engineering.