There is a great post by Harry Pierson over at his DevHawk blog site describing what can we learn from looking at the success of mainstream text-based programming languages to help us in the development of higher abstraction modeling languages that are actually useful.

 

As I have written elsewhere, I am a huge fan of raising the level of abstraction to deal with complexity in our software world.  This is part of what I call the industrialization of software.  No, I dont mean making programming fully automatic, as it will never be that way.  It is too large and complex to do so.  However, I am probably as frustrated as Mini-Microsofts quest to make Microsoft a leaner meaner machine in my quest to make software development more of a predictable and repeatable process.  It seemingly aint going to happen over night and may not happen in my lifetime!

 

John Walker, figured out the industrialization of the engineering design world when his invention, AutoCAD hit the market in 1982.  He said, if you cant model it, you cant build it.  Damn right!  In 2005 and in the software industry, we still have not figured it out, yet.  We are still bashing away with the stone age equivalent of hammers and chisels, whereas the engineering design world has AutoCAD to describe incredibly large and complex building structures, airplanes, engines, electronic circuit diagrams and just about everything else you can think of by modeling blueprints that are meaningful. People use these blueprints to turn models into real world constructs that you and I use every day.  How about that cell phone?  Or your iPOD? Or your car? Or that plane you just flew in on? Or this:

 

 

So whats up with our software world?  Why do we refuse to adapt the successes of other industries, like the engineering design world, and leverage those successes in our software world?  What are we afraid of? Why are we stuck using (still) low level programming languages that we toil with at all hours of the day and night to produce inferior software products?  If we were designing and constructing commodity cars by hand and even fabricating the tools used to build the cars by hand, we would be laughed out of the industry.  How come software development seems to be different?  Where is our AutoCAD for software development?

 

This may seem like a bit of a rant or maybe I have unrealistic expectations as to the maturity level of our software industry.  However, I still cant believe, yours truly after being in the software development business for 15 years, still have to manually add an imports or using source code statement every time I add a reference to an assembly in Visual Studio.  What the ??  Not that I am picking on Visual Studio I happen to think it is an excellent IDE along with introducing Software Factories, Domain Specific Languages, Guidance Automation Toolkits, etc.  While these tools and processes are definitely raising the level of abstraction in dealing with software complexity and advancing the industrialization of software, it still seems not enough to me.  We still lack standards like in the electronics design world for plug and play integrated circuits that I can order from a catalog. Or even standard electronic diagram symbols that describe everything electronic in which anyone trained in the industry can glean, in moments, what the circuit diagram is saying, with no ambiguity.  Are we there yet in our software world?

 

Moreover, we still have a world of programmers that refuse to even consider modeling as a first class software artifact they consider it pretty pictures.  I know we have had issues in the past with CASE tools and with earlier versions of UML and other code generation tools.  But, I have software developers that I have worked with that wont even give it a thought they immediately get out their favorite source code editor and start writing code - so much for design.  And it seems the younger they are, the more I see this behavior or even the old school guys who have become code crafters where they take their craft extremely seriously and are totally affronted in the thought of using a modeling tool.  I am not sure I understand either groups motivation for this behavior.  Where did it come from?  How come I (and a few others) dont have this behavior?

 

Whats my point?  I dont have one I am just pondering, out loud, what it is going to take to bring the industrialization of software into reality and how long?

In previous posts, I discussed, What is a Software Factory, BRIDGEWERX and Domain Specific Languages (DSLs).  DSLs are part of Microsofts Software Factory initiative using model driven development. 

 

This is how Microsoft defines DSLs and model driven development, there is a belief among those involved in software development that somehow, modeling can be applied to make their lives easier. Our vision is to change the way developers perceive the value of modeling. To shift their perception that modeling is a marginally useful activity that precedes real development, to recognition that modeling is an important mainstream development task and not an activity primarily focused on documentation. When models are regarded as first-class development artifacts, developers write less conventional code since more powerful application abstractions can be employed. Model-driven development is thus inherently more productive and agile. Moreover, others involved in development, from business analysts, architects, designers to network staff, and system management specialists, will perceive modeling as adding value to the tasks for which they are responsible. When models span development and run-time activities in this way, communication between people can be optimized, and traceability enabled across the life cycle in any direction. We hold that making modeling mainstream in this way can ultimately change the economics of software development and ensure software systems meet the needs of a business. This approach to model-driven development is part of an initiative at Microsoft called Software Factories."

 

I am a big fan of Microsofts software factory initiative, model driven development and DSLs.  Why?  Because I believe that this approach drives the industrialization of software.  What do I mean by software industrialization?  In my view, it is making the software development process a predictable and repeatable activity, which given the state of the art today, it certainly is not.  Also, as a 15 year veteran programmer, I am looking for anything that will make my life easier in developing software.

 

I believe in the software factory vision so much that Barry Varga and I invented a product called BRIDEGWERX, which is a software factory for generating application integrations.  We built this product using Microsofts BizTalk Server product as the integration engine and developed a Domain Specific Language (DSL) for business analysts and developers to model application integration scenarios using a visual designer, which then outputs a factory schema, which is then used by our software factory template to code generate the resulting application integration solution.

 

While this may seem like a plug for our product (it is , I am also very concerned about the state of the art of software development in our industry.  The last statement in Microsofts definition of DSLs states, We hold that making modeling mainstream in this way can ultimately change the economics of software development and ensure software systems meet the needs of a business.  Isnt that what software is all about?  Meeting the needs of the business? 

 

In any other industrialized industry, such as building architecture, manufacturing, electronics engineering, bridge building, etc., modeling (blueprints) is a considered a first class development artifact.  As the subtext to the title of this blog site states, if you cant model it, you cant build it.  John Walker, the inventor of AutoCAD, figured this out in 1982!  It seems in 2005 we are still trying to figure this out for our software industry.  At least one major software corporation and our small company BRIDGEWERX, is trying to advance the industrialization of software.

There is an interesting web site called lesscode.org.  From their about box: lesscode.org is a place to advocate, discuss, and practice the art of using less code to get more done. We shun complexity and challenge the status-quo when it impedes our ability to simplify our development tools and processes. We appreciate Python, Ruby, LAMP, REST, KISS, worse is better, and talk like a pirate day. lesscode.org is a loose federation of concerned hackers for web preservation and advocacy.

When I first came to lesscode.org, I thought it was going to be about more design and lesscode. However, it seems that the site is about people that really care about handcrafting excellent code. I can respect that, just different to what I though it was going to be about.

 

One of the topics of interest is a debate on scalability and what that means.  I think the issue of scalability is all about the design and has less to do with any particular programming language, application server, framework, or infrastructure. If the design (via the requirements) calls for the application to scale to x, y or z, with meaningful metrics, then it is up to the Architect to design an application, system, or whatever, to ensure the design meets the requirements. And btw, we are in the software world, we can make anything scale nothing that time and money cant solve in our virtual world.

 

However, there is also the issue around making the software development process scalable and one of the lesscode authors made an analogy to restaurants/cooking and talked about different types of restaurant had different scalability requirements.  Other lesscode commenters thought this was a terrible analogy.  MSFT certainly has a view on scalability and they do use the restaurant analogy in which they call a Software Factory:

 

 A software factory is a product line that configures extensible development tools like Microsoft Visual Studio Team System (VSTS) with packaged content and guidance, carefully designed for building specific kinds of applications. A software factory contains three key ideas: a software factory schema, a software factory template and an extensible development environment:

 

Think of the software factory schema as a recipe. It lists ingredients, like projects, source code directories, SQL files and configuration files, and explains how they should be combined to create the product. It specifies which DSLs should be used and describes how models based on these DSLs can be transformed into code and other artifacts, or into other models. It describes the product line architecture, and key relationships between components and frameworks of which it is comprised.

 

The software factory template is like a bag of groceries containing the ingredients listed in the recipe. It provides the patterns, guidance, templates, frameworks, samples, custom tools such as DSL visual editing tools, scripts, XSDs, style sheets, and other ingredients used to build the product.

 

An extensible development environment such as VSTS is like the kitchen where the meal is cooked. When configured with the software factory template, VSTS becomes a software factory for the product family.

 

To press this analogy further, the products are like meals served by a restaurant. Software factory stakeholders are like customers who order meals from the menu. A product specification is like a specific meal order. The product developers are like cooks who prepare the meals described by the orders, and who may modify meal definitions, or prepare meals outside the menu. The product line developers are like chefs who decide what will appear on the menu, and what ingredients, processes, and kitchen equipment will be used to prepare them.

 

How about that? In fact, if you look at any industrialized process it has been designed to scale. The key word being designed. So if the restaurant needs to scale to millions, then design it to scale! While I respect my fellow codecrafters, I am waiting for the industrialization of software to take place so I dont have to keep writing the same base code over and over again for each job that comes my way. After serving a million burgers, you would think the codecrafters would get tired of this too?

Having spent some time in the VS2005 environment, I can say the following about DSLs:

 

A domain-specific language (DSL) is a language designed to be useful for a specific task in a fixed problem domain, in contrast to a general-purpose language. DSLs are gaining popularity in the field of software engineering to enhance productivity, maintainability, and reusability of software artifacts, and enable expression and validation of concepts at the level of abstraction of the problem domain.

 

Using domain-specific languages, one can build customized modeling tools and essentially define a new modeling language and implement it very simply. For example, a specialized language may be used to describe a user interface, a business process, a database, or the flow of information, and then used to generate code from those descriptions.

 

I built a (small) DSL for modeling application integration scenarios, which is always an issue in the IT business world. First, I defined all of the specific domain model terms used in application integrations scenarios such as XML messages, source destination applications, XSL maps, business units, protocols, business rules, etc. Then I described the designer definitions that make up the visualization tool. Once the meta data was defined in a supplied Visual Studio template, you build the solution and another instance of Visual Studio fires up with your visual designer implemented.

 

You then use the visual designer you just created to draw/model the application integration scenario and when you run this solution, it code generates the solution.

 

Of course, I have left out a lot of detail and it was not easy the first couple of times. I have left out a set of code generators, which take a domain model definition and a designer definition as input, and produce code that implements both of the components as output. The code generators also validate the domain model and designer definition, and raise errors and warnings accordingly. But eventually, you get the hand of it.

 

Think of it this way, DSL is a tool for building tools For example, anyone that has used the Class Designer in Visual Studio is using a DSL outputted visual designer, specifically designed for building classes.

 

The process has been quite the learning experience for me and has proven to be very enlightening. I am an old guy, been writing code for 15 years. Quite frankly, I dont give a hoot about which programming language I use cause I see them all as being the same, some better than others, but still hand crafting code. I dont want to hand craft code anymore every time I get involved in a software development project I a) have done it before and b) oh man, this is going to take 6 months of grinding it out. In other words, its gonna hurt.

 

I see DSLs as a evolution in our software industry to use higher level abstractions in the way of visual designers to so that I can spend time designing the solution and have most of the infrastructure code generated for me, that I otherwise would have to grind it out.

 

If you are interested in more information about DSLs and dont mind reading it from Microsoft, download the DSL Toolkit and have a read of the documents at: Microsoft Domain-Specific Language (DSL) Tools

Over the last 15 years as a software professional, I can count on one hand the leaders that I have worked for and that I would work for again.  There are four to be exact.

 

When I was at Eastman Kodak in New York, I worked for Mike Morba who was the VP of Business Development in the Digital Imaging Division.  Mike was responsible for a $30 million budget for Kodaks Picture Center product.  What made Mike a good leader?  First, he knew everything about digital imaging, not only from a software development perspective, but also from a marketing perspective.  He had keen insight into the marketplace and a hell of a vision for the product, which was way ahead of its time.  Mike also knew what motivated people.  It was quite simple really, he gave credit where credit was due.  So when his team did a good job, he made sure people knew about it, not only in his 300 person team, but everyone else in Kodak.  Mike was sincere and selfless in this respect.  He knew he was the man, and so did everyone else at Kodak, but he did not have to advertise it or take credit for what his team did.  A quality other prospective leaders should try and come to terms with, that is, if they can leave their egos behind.

 

Eastman Kodak was a huge company with over 40,000 employees.  At the other end of the spectrum, I worked for Shawn Abbott in a small 10 person start-up company that was more of an advanced R&D shop than a products company.  What made Shawn a good leader?  Not only the attributes mentioned above of Mike Morba, but in addition, Shawn had a vision for his company and never gave up control of his start up, even when times were tough and venture capitalists were hovering around ready to dish out cash but in return wanted control.  Shawns issue was what did these people have in the way of leadership other than cash?  Shawn stayed the course and has done very well for sticking to his vision.  I have a lot of respect for Shawn.

 

Steve Langley is another person I would work for again.  Not only did Steve have the attributes of the people above, Steve was also an extremely focused individual and the first person I met in the software world that said no to ridiculous schedules.  You see, Steve is one of the few people I know who was realistic as to what could be accomplished given the resources at hand.  Steve is a 25 year veteran programmer who knew the value of less code was a good thing, so when the young guys would crank out tons of code, Steve could do the same thing, but much better, with far less code.

 

And the fourth person I would work for again?  Myself.  While I dont think of myself as a good leader, I did start my own software company (because I was frustrated working for leaderless companies, other than the ones previously mentioned) and tried to embody the attributes of Mike, Shawn and Steve for the people that worked for me.  My company that I co-founded with Barry Varga was quite successful for a while.  However, unlike Shawn, I faltered at the choice of taking cash and giving up control, and I found out the hard way that leadership does not necessarily come with cash.  Lesson learned, and when I start up my next company, I wont make the same mistake twice.

 

Whats my point?  Its still the Wild West in the software world where anybody can slap on a set of six guns and call themselves a leader.  Real leaders are extremely hard to find, and in short supply, especially in the nebulous world of software.  Even in the largest software company in the world, the question of leadership is (still) a hot topic with Mini-Microsoft.  Think about the leader(s) in the software company you work for.  Count your blessings if you work for a leader that embodies the attributes mentioned above.  These are the types of individuals that know how to advance the industrialization of software.

Jochen Seemann presented a session on Visual Studio for Software Architects Future Directions in Modeling Tools to about 2000 people at PDC today.  This presentation covered topics of Domain Specific Languages (DSL), Software Factories and Visual Languages.  All of which I am interested in for advancing the industrialization of software.

 

Jochen put into context what a DSL meant by comparing it to an electronics circuit diagram.  In the electronics domain, this circuit diagram has specific meaning, but outside of that domain, it has no meaning whatsoever.  Hence the term, domain specific.  Continuing with the circuit diagram analogy, the language that is used, such as resistors, capacitors, transistors, etc. is specific to that domain.  Hence the term DSL.

 

He also compared DSL to UML where his view was that UML is a broad language for Object Oriented software engineering.  Sure you can customize UML to a large degree, but the problem is trying to use UML to describe something specific enough for a particular domain.  For example, he used the analogy of using use cases (text based) to describe the layout of the auditorium we were in and giving a 50 page use case document to construction workers to build the auditorium.  Using natural language (i.e. English) has a major problem with semantics.  That is, every person could read the use cases, including the construction workers, to build the auditorium and come up with different meaning as to what the words meant and therefore a different auditorium build out.  Hence the problem of using a generalized (read: unified) tool to describe something very (domain) specific.  Of course, no construction worker is going to read a 50 page use case document, they are looking for a blueprint that visualizes specifically what the auditorium is going to look like as the building blueprint is really a domain specific language and is actually a visual language since the blueprint is likely a CAD drawing.

 

This is where DSL comes in. DSL is customizing modeling to the extreme.  Jochen walked through a demo of building a DSL Visual Designer in Visual Studio, which basically consist of three steps.  First you define the domain model, second you define the notation for that model and third map the visualization of the domain model via notation elements.  All of which goes into a large XML schema file.

 

It is an incredibly powerful construct as not only does Visual Studio come with a number of (DSL) modeling templates, but the ability to add third party partner ISV templates and your own templates that all run on the same platform. This makes the interchangeability and reusability with a library of visual designers a reality.  Something that the software industry has been waiting for a long time.

 

Further the ability to link different model aspects and viewpoints means that the ability to code generate complete solutions a reality.  If you are interested in advancing the industrialization of software, I can only suggest that you have a real close look at Microsofts modeling strategy and download the latest DSL tools with the new release available tomorrow. 

The title is a quote from Don Box while presenting Windows Communication Foundation at Microsofts Professional Developers Conference in LA.  Many years ago I read some of Dons books and was always impressed on how he got Windows.  I had the pleasure of seeing him live at this years PDC and saw what a dynamic speaker he is, in addition to the fact that he probably knows more about Windows then any other person on the planet.

 

Dons SOA comment goes directly to how goofy our industry has become where everyone has latched onto this latest TLA.  I cant go anywhere in the software world without running across SOA.  Like Don says, SOA what!  To quote Don, Message, oh message, the truth is on the wire.  There is nothing else.  So much for SOA

 

I was able to catch Steven Sinofskys presentation on Office 12.  The big news is the investment Microsoft has made in SharePoint Version 3.  I thought SharePoint 2003 was pretty cool as that is how I make my living as an Architect, solving customer problems using SharePoint.  However, version 3 is incredible!  It is the center piece of Microsofts Enterprise Content Management (ECM) strategy for the Information Worker.  It represents the 5 pillars of connected systems, Interaction (using Atlas), Messaging and Services (using Windows Communication Foundation), Workflow (using Windows Workflow Foundation), Identity and Access (using InfoCard and AD), and Data (using SQL 2005).  Whats really surprising to me is that at this years PDC, nothing on ASP.NET web application development.  It looks like that SharePoint is the web application of choice.

 

Speaking of Workflow, I got to see Windows Workflow Foundation in detail.  Essentially it is a framework (i.e. a set of APIs) that allows you to embed workflow in your custom developed applications.  Since it is part of the Windows platform, it is available for anyone to use and has built in support in Office 12 and the new version of FrontPage.

 

One really cool feature of WWF is to be able to draw a workflow as a state diagram.  State diagrams are different than sequential workflows as they are more ad hoc in nature and makes a perfect match for many human business processes that are also ad hoc where workflow steps are skipped or even better, dynamic in nature.  Dynamic update meaning we now have the ability to change the workflow logic while the workflow is running.  This is pretty amazing and matches much closer to the real world then a static sequential workflow. This also means that you do not have to recompile.  Cool!

 

Now I have been using workflow (a.k.a. orchestration) in BizTalk 2004 for some time and wondered how the two technologies work together.  As I understand it, using WWF is great for workflow inside applications and BizTalk is great for using workflow across applications.  Makes sense, but I am sure we will see a version of BizTalk that will use WWF in the future.

 

I also wondered how BizTalk 2006 (BTS) will work with Windows Communication Foundation (WCF).  As explained to me, WCF is a great framework for building connected systems (Microsofts term for SOA), but has no facility for orchestration.  In other words, point to point communications.  Adding BTS works in concert with WCF for additional business process and integration server capabilities which adds another layer of abstraction in distributed systems design.  Again makes sense, but I am sure there are other plans for BizTalk in the future which I will get to hear about tomorrow.

 

Also tomorrow, I will get to see and hear a session on Visual Studio Team System for Software Architects and Future Directions in Modeling Tools.  I am quite excited by this as I believe this is another step in the industrialization of software, which is what my blog is all about.  Till tomorrow.

In part 1 of why software industrialization, I explained how even the simplest of end user software products still cant be designed right (i.e. washing machine firmware).

 

I have been a software professional for 15 years and I am still amazed at how little we have improved software engineering in our industry. The software development tools we use still feel like hammers and chisels to me.  The software development process, Agile or otherwise, seems overly complicated, convoluted and hardly predictable or repeatable, in addition to involving way too much effort and using people skills that vary wildly in knowledge and expertise.

 

I gave an example of how AutoCAD revolutionized/industrialized the engineering design world way back in 1982. We have nothing like this in our software development world.  In some respects, I think we have actually gone backwards instead of raising the level of abstraction to improve the process of software development, we are actually writing even more code as products and layers of interfaces are being added to our toolsets daily, plus watching vendor framework class libraries ever increasing in size, (literally 1000s of classes), but not really doing anything to raise the level of abstraction.

 

Here is another analogy I can give you to explain what I mean by the industrialization of software.  Too many years ago, I worked for an electronics company where I designed printed circuit boards by hand.  This meant hanging a circuit diagram (a real blueprint) on a wall and on a light table, with a Mylar grid 4 times the size of the physical circuit board, I laid down red and blue tape (for a double-sided board) of different thicknesses to represent electrical connections between components and different sticky symbols that represented resistors, capacitors, integrated circuit patterns, etc.  Once the layout was complete, we then had to shoot negatives (actually positives) of the layout (reduced by 4X), expose a blank copper plated printed circuit board in a UV oven, with the positive on top and then etched the unexposed copper away in a caustic chemical soup, which all tolled, took 2 weeks to complete.

 

I would work 8 hours a day times 6 to 8 weeks staring at a light table and as I made each connection (out of several hundred), I would highlight the connection on the circuit diagram (i.e. blueprint) to indicate a completed connection.  There were several design constraints, layout size of the circuit board, making sure power traces were not beside low level signal traces, etc.  Kinda like a chess game, except it took 6 weeks or so to see if you won or not.  If you painted yourself in a corner, it invariably meant a starting from scratch.  Management did not appreciate starting over, so we got real good at playing chess

 

Then industrialization hit the electronics/printed circuit board industry with the evolution of computer numerical control (CNC) that helped digitize the circuit diagram into an autorouter software program that automatically designed and laid out the printed circuit board from the digitized input by a light pen.

 

 

 

Today of course, this manual labor intensive process is fully automated.  The circuit diagram is now fully digital (remember AutoCAD) and the autorouter can simply read the circuit diagram and produce the circuit board directly with no human intervention or other intermediate steps.  In fact, the electronic circuit diagram is fed directly into this system and out comes the finished printed circuit boards (forget double sided, how about a dozen layers!), all in a matter of minutes to a few hours.  What used to take me 6 to 8 weeks or longer by hand is now reduced to minutes or hours, with no errors. How is that for industrialization?

 

What can we point to in our software industry, over the same time frame as our analogy above, that has reduced two months of manual labor intensive hand-coding effort into a matter of minutes or hours?  And I dont mean marketing words or the latest TLA buzzword SOA, or latest programming language, I mean some tool or generator that actually reduces the manual effort and coding required to produce a finished software product.  Can anyone point to anything that substantially has advanced the industrialization of software over the last 15 years?

 

In the software industry, we are still in the dark ages as far as I am concerned and my little blog here is a small attempt to try and help push the envelope of software industrialization from an educational perspective.  Hope you enjoy it.