# Sunday, 27 April 2008
Aeroplan – A Modern Web Site is Down for 2 days!

OK, I promised I would be good.  At the beginning of this year, I made myself a promise not to be a cynical “you know what” when it comes to the world of software development.  Lord knows I have seen my fare share of Dilbertesque wackyisms over the last 17 years in this biz.  Heck, I even contributed back to the industry with my own open source project.  I felt it was giving back to the software world, that has been so kind to me, but then again maybe it is some form of redemption.  I don’t know.  Maybe that is why I have had only one post so far this year.

 

But… there are still some things that defy any sort of rationale, logical explanation whatsoever.  For example, I am a frequent flyer (~25,000 miles per quarter) and so I keep pretty close track of those miles on my Aeroplan.  Last week, I got back from a business trip to LA and Medford, Oregon.  Oregon, truly a beautiful place, but not as beautiful as the Sunshine Coast, where I live, but I am biased.

 

So, as usual on Friday, I pop over to the Aeroplan site and am greeted with the following web page.

 

 

 

 

Now, I know from time to time, there are outages, and maybe it would last an hour or so, cause how serious could it be?  Pop in a new drive, slide in a new web server in the farm, switch failed, power supply failed – but hey, even those would actually not cause an outage, cause as any data center system engineer knows, these are all “hot swappable” in the year 2008. Well it is Sunday at 9pm and the outage is still in effect.  Now what in the heck would cause an outage of a fairly heavy traffic laden web site for over two days?  I can’t imagine.

 

Note the site says, “Following a routine maintenance procedure, we experienced a system failure.”  A system failure?  Folks this is a catastrophe of biblical proportions in web terms for the year 2008.  A rip in the time space continuum has occurred over there at Aeroplan.  No halfway decent web site goes down for two days.  Even my own rinky dinky web log here that I host on an ancient 1990 Pentium III 600 MHz piece of you know what has not even seen 2 days of outage – ever - even when my power supply blew up and I had to shoe horn the ol’ mother board into a brand new case only took 6 hours.

 

So what the heck could have happened?  I can only guess that somehow the entire data center melted down and there is no back up of the source code and somewhere someone, or a group of people are madly waiting until Monday morning to get into a bank’s safety deposit box to get the latest tape back up.  Like I just can’t image what it could be.  Can you?

 

What will be really surprising to me come Monday morning and Aeroplan is still not up.  If that is the case, I should call Aeroplan – I will host their site for a mere $10,000 per month (cheap!) and I guarantee that their site will never be down for longer than 24 hours, or they can have all of their money back.  How about that Areoplan, do we have a deal?  Oh by the way, I might upgrade my PIII to host it on :-) 

 

But seriously, man, you are giving us professionals a bad name in our biz for not having your sh*t together.  There is absolutely no excuse whatsoever, in the year 2008, for your main business, i.e. your web site, to go down for 2 days!  Either you do not know what the heck you are doing or you just don't care Mr.Aeroplan - either case, if you did not "own" my aero miles, I would switch to a new provider in a flash for such blatant shoddiness.

 

Updated Monday April 28 - Aeroplan manages to get their web site back up and running with little explanation.

However, I still see that my Aeroplan miles I collected from the previous week have not been updated on the site.  We will see if my Aeroplan Miles were affected or not...

Failures
Sunday, 27 April 2008 20:43:52 (Pacific Daylight Time, UTC-07:00)  #    Comments [0]
# Wednesday, 28 December 2005
Top Software Failure for 2005 or The Future of Software
"I did not get what I envisioned" from the project, the senior official acknowledged. But he said the F.B.I. today had a better understanding of its computer needs and limitations as a result of the effort." The lesson we have learned from this $170 million is invaluable," he said.

This quote is from a $170 million failed software project in the US called the Virtual Case File or VCF by the FBI.  The system was designed to give the bureau's nearly 12,000 agents around the country instant access to F.B.I. databases, allowing speedier investigations and better integration of information both within the bureau and with other intelligence agencies that must coordinate national security matters.
 
While there were several software failures for 2005, I chose this one to highlight a point.  The very first sentence, I did not get what I envisioned is a telling statement that demonstrates a fundamental reason why software projects fail.  There is an incredible amount of information written about why and how projects fail that has been well documented for dozens of years. The following reasons for software failure are a summary: poor user requirements, poor project management, no change control, too much complexity, unskilled staff, political issues, poor software development processes, and on it goes.  So if these reasons are well documented over the last 20 years, why do we still have failed software projects?
 
I would say we have a more fundamental problem than all of these reasons combined and that is we have no way of visualizing the finished software product before it is built.
 
In other industrialized engineering disciplines, we have many ways of visualizing the end product without actually building it first.  For example, models are used extensively to visualize cars, buildings, cell phones, iPODs, bridges, airplanes, trains, you name it and somewhere there is a model that visualizes what the finished product looks like.  The model may be an artists rendering or a precision built to scale model of an airplane.  The fact remains that no matter what the product is, there is a visualization of some sort that people can look at as a concrete vision of what the end product looks like before it is built.
 
What do we have in the software world to visualize what the finished software is going to look like?  Not much.  Some people would call prototyping or screen shots or using Visio as a way of visualizing software.  Unfortunately, none of these represent what the actual finished software is going to look like.
 
As a software professional, whats most embarrassing about the VCF failure and other failures is that 95% of all business software applications are nothing more than retrieving data out of a database, presenting it to a human, who may manipulate it and saving it back to the database. Thats it.  We do it over and over again. No matter if you are SAP or salesforce.com or a custom application in the business world, all that is being done is retrieving and saving data to a database, yet billions are spent each year doing this and if you believe statistics, over 50% of these projects will fail.
 
It would seem to me that the software industry is ready for some industrialization lessons learned in other engineering disciplines, with the first lesson being lets visualize what the end product is going to look like before we build it.  Here is an excellent example of a visual model that was created 100 years ago to visualize an immense and complicated structure.  To me, this model puts into perspective how little we have evolved in the software industry, with all of our technology we still cant build software right.  Maybe next year
Failures
Wednesday, 28 December 2005 12:52:46 (Pacific Standard Time, UTC-08:00)  #    Comments [0]
# Wednesday, 27 July 2005
The sorry state of our software industry is there hope?
It is no wonder to me why our software industry is in serious trouble.  FBI officials said they hope to award a contract by the year's end for a complex new software program to replace a failed project that was canceled this year at a cost of more than $100 million to taxpayers. 
 
Closer to home, the Canadian government has spent over $2 billion (yes, $2 billion!) on a gun registry that is nothing more than a giant database for storing information about legal owners of guns who have volunteered to register their weapons.  But $2 billion dollars? To coin a Dave Barry phrase, I am not making this stuff up.  It really happened.
 
The software that controls the baggage handling at the Denver airport where, due to a bug in the software, it was costing the airport $1.1 million dollars of lost operational cost per day!  At the time this was occurring, there was no predicted end in sight.
 
Finally, have a look at The CHAOS Study which reveals mind boggling statistics on software development failures.  The report summarizes that in 1995, the U.S. spends $250 billion per year on information technology for 175,000 software projects.  31% of those projects were canceled before completion. 53% of those projects cost 189% of original estimates.  $81 billion was spent on canceled projects.  Only 16% of software projects were complete on-time and on-budget.  16% wow!
 
Some of you may point out that this report is 10 years old.  I can tell from my own experience, and others, not much has changed since then.  In fact, an updated report verifies this.
 
For the experienced software developer, this does not come as a surprise.  Disheartening yes, surprising, not really we live it - everyday.  The CHAOS Report states that the top three causes for these failures are the lack of user input, incomplete requirements and changing requirements. No surprise here either, however, I would like to offer up what I would call the root cause of these failures, that is not categorized in the report, which is totally underestimating the size and complexity of any software development effort.
 
Software development is an incredibly complex, highly skilled, maximally labor intensive process, probably more so then any other professional human endeavor. Software design and programming is so complex and error prone that it is totally underestimated by everyone, including the majority of software programmers who are closest to truly understanding this size and complexity issue.  Further, the people that we are trying to develop the software for have no idea about size or complexity or how software is designed, built or even how it works.  It is a complete mystery to almost everyone.
 
Tomorrow we are going to delve deep into this size and complexity issue. 
Failures
Wednesday, 27 July 2005 04:06:06 (Pacific Daylight Time, UTC-07:00)  #    Comments [0]

© Copyright 2010 Mitch Barnett - Software Industrialization is the computerization of software design and function.

newtelligence dasBlog 2.2.8279.16125  Theme design by Bryan Bell
Feed your aggregator (RSS 2.0)   | Page rendered at Saturday, 07 August 2010 02:43:18 (Pacific Daylight Time, UTC-07:00)