 Wednesday, January 11, 2006
In part 3, I discussed using storyboarding as a way to help industrialize the software development process. Business users interact with software 100% of the time through a user interface. If we as software developers focused our effort developing storyboards for a business software application, this would result in a complete specification of what the business user wants without writing any code. It also dramatically reduces the tremendous gap between requirements (i.e. intent) and executables (i.e. what is delivered). Further it industrializes the way we develop software from a mostly trial and error approach to one that is repeatable and predictable, thereby reducing project/product risk substantially, reducing the cost of the project and minimizing schedule overruns.
This storyboard approach is not novel as it is tried and trued in the film industry. Whats novel is that it is rarely done in the software development world. As software developers, we spend so much time under the technical covers of applications we forget that business users do not care, the business user is only interested in their data as represented in a user interface and what they can do with that data.
Imagine if we had a catalog of storyboards (i.e. models of user interface screens, not implementations) based on vertical business industries, such as a CRM application for manufacturing or a supply chain application for electronics distribution or a workflow application for order processing or Lets say we had a catalog of a thousand storyboards that we could flip through and select storyboards for our customers for a specific business application. We could go even further by saying that these storyboards contained metadata that describes the user interface elements that make up the storyboards. Based on the selected storyboards for a particular application, we could code generate a specific technology implementation of those storyboards, from the storyboard metadata, which means transformed into an actual user interface implementation for a specific targeted technology.
Pipe dream? Maybe. On the other hand, I see the current way we are developing software as a pipe dream or at the very least it requires many development iterations spending lots of effort and money to produce something that does not necessarily meet the business users requirements. It currently takes too long to develop a commercial grade business application of any size or complexity. Remember the dot com era? Customer got fed up en mass with our software industry that set unrealistic expectations of what software can do. Customers will likely get fed up again, sooner than later. Corporations are also fed up spending millions ( ok, billions) on software development projects with seemingly no end in sight and at a 50% failure rate you would think there would be a revolt. If I was a customer, I would say the failure rate is unacceptable. Some would say the odds are better in Vegas. Something is going to give. It is this process that is going to drive the industrialization of software.
User interface storyboards contain metadata that describes the user interface elements in a non-implementation specific way. As mentioned before business users do not care about which technology, they care about how fast and cheap the implementation can be done. Since an implementation of a storyboard (or set of storyboards) could be code generated, the selection of which implementation technology could be based solely on what IT infrastructure exists already in the company, thereby leveraging the companys current assets.
While we dont have a catalog of storyboards today, it would not be that difficult to assemble since most if not all the user interface screens, from an implementation view, exists today in the tens of thousands of applications that are already in the market, albeit some better than others. We could abstract these implementations into storyboards, including the metadata that describes the user interface elements, and start assembling a catalog of user interface storyboards.
Lets assume we can do that for just a moment. Lets say we have chosen 50 storyboards out of a catalog that represent a particular business application and we have code generated an implementation of those storyboards into a technology specific implementation. How do we hook those user interfaces up to data? How do we do transaction processing? What about workflow? Business logic? And why do we abstract at the user interface layer? These are topics for my next post in this series.
Friday, January 06, 2006
In part 2 of this series, I discussed the fact that a business user of software spends 100% of their time interacting with the user interface, yet we as software developers, spend only 10% to 25% of our time developing user interfaces and spend most of our time under the covers, buried in the technical details. As mentioned in the previous post, the business user could care less about whats under the covers in any given software application.
I suggested one way to increase our software development productivity and reduce product delivery risk is to spend more time on user interface development and the interactions with the business user. I suggested that we borrow a tried and trued technique from another industry, the film industry, and develop inexpensive storyboards, that represent the user interface screens that the business user actually interacts with.
While we do use some storyboarding techniques in our software development process, we dont take it far enough. I would suggest that no code is written until all of the storyboards for a business application have been developed and approved by the business user(s). Much in the same way all of the storyboards for a film have been developed and approved before a single frame of film has been shot.
I think that following this storyboarding approach (without writing any code) would dramatically increase the success rate of designing and constructing business software applications. This would remove the trial and error approach (read: major source of risk) to software development that we currently have today or at least for business software, where the business user spends 100% of their time interacting with the software. Some might say that prototyping (in iterations) is the same approach. I would disagree, because prototyping involves writing code and soon we are lost in the details and mechanics of writing code, (under the covers again) and increasing the gap between what the business user wants in a software application and what we think the business user wants. Remember the business user interacts with the just the user interface 100% of the time.
Others would say that users dont know what they want so how can we develop the storyboards? I would suggest that if the business users dont know what they want, then we have no need to write a software application. While I am being a bit facetious, I also believe this to be true. If the user requirements are so ambiguous that storyboards, even using pen and paper, cant be developed, then whats the business need for software? It has been my experience that someone or persons in an organization knows exactly what is required, you just need to find those people.
So how do we storyboard what a business user wants? First we need a storyboarding tool to do this. In the film industry, it is still done mostly the old fashioned way pen and paper. However, since we have a computer and the business user is the target user of the storyboards and the storyboards themselves are the representations of user interfaces and provides the interaction between the business users, it makes sense to develop the storyboards as complete (representations of) user interface screens.
At one point in my career, I remember working on a project where we took this storyboarding approach and developed all of the user interface screens and the order that they were used for business users, using Windows Forms in Visual Basic 6. In fact, a few of our programmers at the time, could literally whip the forms up on the spot as they were very adept with VB6 and with absolutely minimal coding could storyboard the order of the screens while clicking on OK or next buttons.
The business users were ecstatic with this approach cause within a few weeks we had developed over 50 screens that represented the entire business application. Not only did this give the business users exactly what they wanted, it also provided us with a complete blueprint of what was to be constructed. This dramatically reduced the risk of the project failing and reduced the cost of the project because we had our complete specification. This is one approach to industrializing the software development process.
Next we will explore a specific storyboarding approach that separates user interface definition from implementation.
Thursday, January 05, 2006
In part 1, I discussed the exponential pervasiveness of software in our world, yet to non-software professionals (i.e. programmers), the process of creating software is highly skilled, manual labor intensive process, that is still mostly trial and error. Due to this trial and error process, 50% of most software projects/products undertaken will fail. How can we have all of this incredible software yet the process of creating it is still in the dark ages?
Throughout this blog are several examples and references of failed software projects and the symptoms of the reasons why they fail. The bottom line is that the software development process has yet to be industrialized into a predictable and repeatable process.
How do we industrialize software development? A lot has been written on the subject over the years, yet from where I sit as a 15 year software professional, none of the methods or processes suffice. Why? I think it mostly has to do with not looking at software development strictly from a users point of view. For example, consider a typical business user of software who comes to work everyday and launches up their CRM application or any other business application. What is that business user mostly concerned with? Their data represented on the computer screen, usually in a form or graph or spreadsheet or report of some sort. The business user wants to input, manipulate, search and see their data in a manner they can understand and work with. The business user assumes that the data is accurate and valid when they are working with it. And when they are done working with it that it is stored somehow and can be later retrieved exactly the same way it was stored. The business user also wants the software to work the same way every time they use it.
What the business user does not care about is the technology used or how the data gets to their computer screen. They also dont care about how many hours, weeks or months (or even years) it took to make this work. They just want their data now and in the format they want. As John Crupi says, "Biz folks don't care if you use two cans and a string to help them get to market faster and cheaper" 
So what does this mean for industrializing the software development process? It means spending time on the piece that means the most to the business user which is the user interface. Thats it. Thats all the business user sees and interacts with. We, as software developers, spend inordinate amounts of time learning and using new programming languages (like C# 2.0), new architectures (like Service Oriented Architectures), new development methodologies (like Agile) and new operating systems (like Windows Vista), but yet, none of this has little to do with what the business user interacts with on the computer screen.
As mentioned above, the business user spends 100% of their time interacting with the user interface on the computer screen. Therefore it would make logical sense that, as software developers, we spend more time focused on this area then anything else. Sadly, in most of the projects I have worked on, user interface design and construction is but a small part of the overall development effort. I would say that it represents maybe 10% to 25% at most of the total effort. I sometimes wonder why this is. Further, we usually use words (as in UML use cases) to describe the user interface and interaction with the user. Funny thing is the user does not interact at all with these words and could care less. Where are my screens they say.
I think one of the ways to industrialize the software development process is to focus more effort on the way a business user interacts with the software. As software developers we could learn a lot from another industry that has figured this out, which is the film industry. Before any large amount of money is spent in the actual making of a film, detailed storyboards are developed that layout the complete film before a single frame is shot. So why dont we use storyboards in our software development process? This will be the subject of my next post.
Sunday, January 01, 2006
Computers and software has become so pervasive in our daily lives it is hard to imagine doing anything without it. In fact, virtually everything we touch daily is powered by a chip of some sort and we as humans interact with the software running on it. Whether it is our cell phone, iPod, digital camera, television, video, car or the software we use at work and play, software is everywhere. With the price of computers continually dropping and the power continuously increasing, coupled with exponential growth in internet connectivity, there is simply going to be more software in 2006 and for years to come.
However, a fundamental problem still exists in the world of software. The process of creating software is still a labor intensive, error prone process that is fraught with complexity, bugs, schedule/cost overruns and unhappy users. To non-software professionals who are the masses that make up the software user community, software seems to be doing ok - maybe. I can listen to my iPod, while getting email on my computer and seeing text messages come in on my cell phone. However, what the masses dont see is the incredible effort required to design, construct, test and release any type of commercial software that has minimal bugs and meets your requirements.
As a software developer, the process we use to design and construct software is still trial and error at best. It may come as a surprise to non-software professionals that there isnt a predictable and repeatable process for developing quality software on time and on budget. While there are hundreds of books and thousands of articles on how to design and construct software, they are all different, with no tried and true process. Of course there are commonalities, but as the saying goes for every user requirement, there are a thousand designs and for each design there are a thousand ways to implement it. The variability in how we develop software is the curse of our industry.
Over the last 15 years as a software professional, I have worked on many software projects and products, some small, some very large and a lot in-between. I have also worked with hundreds of people on these projects. As I reflect on those projects, I realize that the ones that succeeded were based mostly on the highly skilled and competent people working on these projects.
Perhaps another surprise is that a lot of these people did not have Computer Science degrees, but rather came from all walks of life, however, all of these people had one thing in common they knew intuitively how to design and construct software. While some of this can be attributed to experience, most of it was an innate ability to read words as user requirements and translate those words into a working software program that somehow met what the user (community) wanted and was done on time and budget.
So what does this mean? It means that the design and construction of software, particularly business software, is still an unindustrialized process. What do I mean by unindustrialized process? I mean that we have progressed very little in the way of automation. We still hand-craft software just like cars were handcrafted before the concept of producing parts that were then assembled. That was almost 100 years ago!
It still amazes me that we have this incredible technology called software, yet our process of creating this software is quite archaic. On the industrialization scale, compared to other industrialized disciplines like manufacturing, building construction, electronics, etc., we are decades behind. What will drive industrialization in the software industry? This will be the topic of my next post in this series.
Wednesday, December 28, 2005
"I did not get what I envisioned" from the project, the senior official acknowledged. But he said the F.B.I. today had a better understanding of its computer needs and limitations as a result of the effort." The lesson we have learned from this $170 million is invaluable," he said.
This quote is from a $170 million failed software project in the US called the Virtual Case File or VCF by the FBI. The system was designed to give the bureau's nearly 12,000 agents around the country instant access to F.B.I. databases, allowing speedier investigations and better integration of information both within the bureau and with other intelligence agencies that must coordinate national security matters.
While there were several software failures for 2005, I chose this one to highlight a point. The very first sentence, I did not get what I envisioned is a telling statement that demonstrates a fundamental reason why software projects fail. There is an incredible amount of information written about why and how projects fail that has been well documented for dozens of years. The following reasons for software failure are a summary: poor user requirements, poor project management, no change control, too much complexity, unskilled staff, political issues, poor software development processes, and on it goes. So if these reasons are well documented over the last 20 years, why do we still have failed software projects?
I would say we have a more fundamental problem than all of these reasons combined and that is we have no way of visualizing the finished software product before it is built.
In other industrialized engineering disciplines, we have many ways of visualizing the end product without actually building it first. For example, models are used extensively to visualize cars, buildings, cell phones, iPODs, bridges, airplanes, trains, you name it and somewhere there is a model that visualizes what the finished product looks like. The model may be an artists rendering or a precision built to scale model of an airplane. The fact remains that no matter what the product is, there is a visualization of some sort that people can look at as a concrete vision of what the end product looks like before it is built.
What do we have in the software world to visualize what the finished software is going to look like? Not much. Some people would call prototyping or screen shots or using Visio as a way of visualizing software. Unfortunately, none of these represent what the actual finished software is going to look like.
As a software professional, whats most embarrassing about the VCF failure and other failures is that 95% of all business software applications are nothing more than retrieving data out of a database, presenting it to a human, who may manipulate it and saving it back to the database. Thats it. We do it over and over again. No matter if you are SAP or salesforce.com or a custom application in the business world, all that is being done is retrieving and saving data to a database, yet billions are spent each year doing this and if you believe statistics, over 50% of these projects will fail.
It would seem to me that the software industry is ready for some industrialization lessons learned in other engineering disciplines, with the first lesson being lets visualize what the end product is going to look like before we build it. Here is an excellent example of a visual model that was created 100 years ago to visualize an immense and complicated structure. To me, this model puts into perspective how little we have evolved in the software industry, with all of our technology we still cant build software right. Maybe next year
Wednesday, December 21, 2005
If you have read this 7 part series, you will see how our DSL Visual Designer works for declaratively defining complete application integration models and see how our Software Factory works by taking the DSL Visual Designer tools output (an XML model definition) and using it to configure our Software Factory template, which is a set of scripts that configures a Visual Studio solution, including BizTalk server as the core integration engine, to code generate complete application integration solutions. Our framework design pattern can be applied to any application integration scenario. Further, I assert that the Software Factory approach can be applied to any software problem domain just as successfully as we have applied it to the application integration domain.
Our DSL Visual Designer is very specific in the sense that its domain language is specific to application integration scenarios. We built our Visual Designer on top of BizTalk Servers integration engine to provide a higher level abstraction so that Business Analysts (BAs) can easily model their specific integration scenarios as BAs typically have the business domain knowledge and just need a modeling tool, specific to that business problem domain, to model the solution which is declaratively complete. Since the model is complete (i.e. high fidelity, meaning not missing any pertinent information) and that we are in the virtual world of software, we can code generate the entire output, meaning producing the solution in a predictable and repeatable way.
Thats the main point of software industrialization. As mentioned in the introduction to this series of blog posts, software industrialization means raising the level of abstraction to producing software solutions in a predictable and repeatable manner. Producing software solutions in a predictable and repeatable manner is the bane of our industry. In our case, using the Software Factory approach enables us to use model-driven development techniques using Domain Specific Languages (DSL) and assemble pre-built components based on a domain model pattern specifically designed for modeling application integration scenarios.
Because we designed and constructed over 25 application integration scenarios by hand over a course of 4 years we saw many application integration patterns based on producing these solutions for several different industry verticals and different types of integration scenarios (i.e. basic messaging exchanges to complicated, recursive workflows, all using orchestration). In fact, in almost every project we were seeing these patterns repeat themselves over and over again. Based on these repeating patterns (i.e. our evolution), we saw how we could embody these design patterns into a modeling tool (i.e. our DSL) and take the modeling tools output (XML definition of an application integration scenario) and configure a Software Factory to code generate the output, producing an installer package that can be easily installed on the target system, including a number of runtime applications that can monitor and manage the installed application integration solution.
I personally believe that this Software Factory approach will be the way that most (business) software applications will be designed (i.e. DSL models) and constructed (i.e. code generated) in the future. Why do I say future? Why has this approach not caught on (yet)? This blog has discussed many themes to the reasons why. Mostly it boils down to education. I work with developers today that do not see the value in interface-based programming let alone the value of model-driven development or code generation or the concept of a Software Factory.
The education/experience gap in our own software engineering community is quite large and when some programmers (new and seasoned alike) are introduced to these concepts well, lets say that there is considerable resistance to even thinking about using this approach.
Now throw into the mix the vendor community that insists on producing marketecture. Our very own software vendors are doing us a disservice by selling hype to our target business community. Now add the business community to the mix, who are the target audience of this marketecture to most, software is a complete mystery. All they know that it is an incredible time consuming, costly, painful process that mostly results in never ending work, endless upgrades and broken promises. Unfortunately, given the current state of the art in the software industry, this is all but true.
With a few exceptions, I consider the software engineering industry to be still in the dark ages, yet to go through the industrialization process that other, more mature, engineering disciplines have evolved to. This series of posts is intended to show, in one companys way, how we evolved the industrialization of software in the application integration domain using modern software engineering techniques and tools.
Tuesday, December 20, 2005
Parts 1 through 5 discussed the details of our application integration framework design pattern. A Business Analyst (BA) uses our DSL Visual Designer (as a RIA) that has our framework design pattern implemented to declaratively model specific application integration scenarios. There is one optional point in the declaration where a small amount of custom code may be required and that is if a custom application adapter is required. Other than that option, any application integration scenario, regardless of size or complexity can be 100% declaratively modeled using our DSL Visual Designer.
This goes directly to the point of why model driven development using the Software Factory approach works which in my opinion, represents the state of the art in software engineering today. My intent in writing Parts 1 through 5 was to walk through how our application integration framework pattern can be used to model any application integration scenario. The output of our DSL modeling tool is an XML definition that declaratively defines a specific application integration solution as defined by a BA and validated by our Software Factory schema (i.e. our framework model that we just walked through in parts 1 through 5).
Next I will discuss our Software Factory in detail and how it works based on a particular application integration definition. Have a look at this logical view of our Software Factory. It contains 3 subsystems, they are: the DSL Visual Designer that we just walked through, our Software Factory, and our runtime system.
You already know what the DSL Designer does. The Software Factory takes the DSL Designer XML tools output and configures our Software Factory template. This means that all of the properties that were set by a BA in the DSL Designer are interpreted by our Visual Studio script and populates a Visual Studio solution with pre-built code artifacts, including source and destination XSD schemas, adapters, business rules, XSL maps, any custom assemblies uploaded by the customer, source and destination namespaces, BizTalk orchestrations, and our pre-built runtime applications are added to the solution.
Once the Visual Studio solution and projects have been populated with the specific XML configuration information, a deployment project is added to the solution and the solution is built and packaged as a .msi, ready for deployment. The Customer is then notified that the package is ready to be downloaded, the Customer downloads the packaged solution and runs it on the target server. During installation, we ask the Customer for credentials for the solution to run along with the source and destination addresses of where the applications to be integrated exist on the network. In our .msi, we check for the existence of BizTalk Server, SQL Server, IIS, etc., to ensure all of the services are installed and running correctly.
Configuring and extending Visual Studio is a relatively simple process as the Visual Studio extensibility model is well documented and provides for easy automation. After designing and developing 25 BizTalk integration solutions, setting the same old properties, adding pre-built orchestration schedules, schemas, etc. is easy. Sure, some projects are more difficult than others and a little manual manipulation may be required, but generally the time it takes to configure and build an application integration solution using the Software Factory approach is a matter of days instead of months, and thats the point of this series of posts.
Next post we wrap up our discussion on the evolution of software industrialization.
Monday, December 19, 2005
In part 4, we discussed our application integration framework design pattern at the level of the Connection object. A Connection represents a logical connection to each Application, and physically to the Applications Adapter. A Connection contains 1 or more DataFlows. A DataFlow is a collection object that contains 1 to 4 schemas (i.e. XSDs), 0 or more rules, and 0 to 2 maps depending on DataFlow communication type.
With respect to rules, for anyone that has used Microsofts Outlook Rules and Alerts functionality will recognize our GUI Rules Designer. As mentioned in another post, its been done before, you just have to go look for it . The Rules Designer contains a Rules Manager that allows the BA to add, edit or delete rules in an ordered list, a Rules Editor that allows the BA to declaratively make rules, A Fact Editor that allows a BA to declaratively name a rule, and A Rule Expression Editor that allows the BA to define expressions, and if required, provides the facility to upload custom rules (i.e. .NET assemblies).
This diagram shows a screen shot of our Rules Designer:
At this level we also provide the capability for BAs to upload test XML documents that can be validated against the specified schemas. The BA can also test the (XSL) map for proper transformations. Any errors for schema validation and/or map transformation are returned to BA in the Designer. In fact, if the BA provides a valid XML document with test values that represent what the real values are likely to be, business rules can also be executed, in which the return values are the resulting actions that would occur for the rule processing.
Thats it. The BA has now declaratively configured a complete application integration scenario using our DSL Visual Designer tool that implements our application integration framework design pattern. As mentioned before, our Visual Designer is exposed over the internet as a Rich Internet Application (RIA). This allows a BA easy access to their application integration solutions from anywhere in the world, in a secured manner. This approach models the software as a service scenario where there are no requirements for an organization to use the Visual Designer other than a browser.
Now what? After the BA has completed defining the application integration model and saved the model, the BA can now order the integration definition to be built (i.e. code generated) from our Software Factory. The order includes pricing based on the number of application integration connections modeled in the scenario. Once the order has been placed, the turnaround time could be a few hours to a few days, depending on options ordered. The BA then returns to the Visual Designer web site and can download the resulting MSI to be installed on their target system. Once installed, the BA and/or System Administrator can run a self-test to verify and certify the application integration solution for correctness.
The runtime installation also includes other tools to manage and monitor the application integration solution. This includes Business Activity Monitoring for monitoring the application integration scenario, a Centralized Exception Manager that handles any runtime messaging exchange exceptions, complete with notifications, and an Integration Manager that allows runtime configuration of security credentials and application end point locations.
With respect to making any changes to the application integration scenario, the BA can launch the Visual Designer again, load up the existing model, that has been versioned, make modifications (to a schema for example) and save, then order and get another MSI that performs the upgrade process on the installed application integration solution.
Next post I will discuss our Software Factory for taking application integration scenarios based on our framework design pattern (as realized by our DSL Visual Designer and output as an XML definition) and walk through the steps on how we configure our Software Factory template to code generate the specified solution.
© Copyright 2008 Mitch Barnett - Software Industrialization is the computerization of software design and function.
newtelligence dasBlog 1.9.6264.0 Theme design by Bryan Bell  | Page rendered at Saturday, October 11, 2008 7:36:20 AM (Pacific Standard Time, UTC-08:00)
|
|