- don't forget to click on  on bottom right of player

 

Recently, I have been focusing on updating my web-based DLR IDE which is one of five components that make up a larger project called Global System Builder.

 

In a previous post, I mentioned that I would like to develop a DLR IDE in Silverlight, but I am still having a time figuring out Silverlight, especially trying to debug web-based Silverlight applications.

 

In the meantime, I continue to extend this JavaScript code editor called “EditArea” by Christophe Dolivet.  What an excellent job he has done!  Features include:

 

·        Live syntax highlighting for several languages and the ones that I am interested with the DLR which is Python, Ruby and Managed JScript.

·        Multiple-document support with tabs

·        AJAX load and save functions

·        Line numerations

·        Search and replace (with regexp)

·        unlimited undo and redo

·        Auto-indenting – great Python support!

·        Font resizing

·        Internationalization

·        Toolbar customization – add your own commands via plug-ins

·        Full screen capability

·        Create new syntax or language files

·        Multi-browser support

·        Multiple instances on the web page (if required)

 

Really quite amazing and it has been working excellent for me.  Great job Christophe!

 

I can load and save files to the disk in a project hierarchy.  I can create new files and then save to disk.  If you open a file and start editing it, you will notice that an asterisk is added to the filename tab to tell you that the file has changed. 

 

It looks like there are two items left for me to implement, each one with their own set of difficulties:

1. Intellisense
2. Debugging

I prefer to GUE (Go Ugly Early) to try and mitigate risks, so it is going to be Adventures in Debugging.  But before I do, Intellisense will be tricky as well since Intellisense has quite an array of features.  My goal is to only implement “List Members” and “Parameter Info.”

 

Back to debugging and without getting too philosophical, I can’t image any “real” IDE that does not have debugging support, especially if you are working on any size of code base.  I view it as the same as driving at night with no headlights – you are bound to hit something hard sooner than later. So what choices do I have for debugging with the DLR and my custom IDE?  As far as I can tell, there appears to be one choice and that is MDbg.

 

You can download MDbg and compile it.  You will need to add this trick (see answer at bottom) to get the help to work on the command line.

 

Mike Stall is the MDbg guru and he has a great MDbg link fest to get you started.  I also really liked Jan Stranik’s Introduction to the Managed CLR Debugger as it shows how you can easily write an extension using the debugger API.  You can also download the videos.

 

So, how to start?  I started off by running MDbg and then issue the command line “load gui” to load a Windows GUI extension to make it easier to use then the command line.  This extension provides an excellent example on showing how you can get a debug window to step through code, show locals, etc.  Here is the code that I want to debug in my editor:

 

 

Dead simple, but a place to start. On the web server, I have a C# Windows Forms application that hosts an IronPython 1.1.1 engine in it.  I also have IP 2 with the DLR prototyped, but just trying it out on IP 1 for now. The application hosting the IP engine is called GSBService1.exe, compiled with debug symbols.  I can run MDbg and launch the app in the debugger:

 

 

The debugger attaches to the running process and breaks into the code at the main entry point:

 

 

You can type help to get a list of commands available that allows you to do all sorts of things such as setting/clearing breakpoints, stepping in and out of code, etc.  In addition, from the tools menu, you can open other windows such as viewing the Callstack, Locals, Modules, Threads and QuickWatch.

 

One of the commands is “x” which lists the loaded modules.  You can see which modules are loaded in my application.  You can also specify a particular module and display specific functions by pattern matching.  Here is the help usage:

 

 

Note that I am looking for the function in the module gsbservice1 that executes my .py file – I know the function contains “ParseInteractive”, so I use the x command again to find it using the pattern matching.  Then I can specify a break point by the command b ~1.  I then issue the go command (g) and now I am running and all set to debug at my breakpoint.

 

I go back to my web-based code editor and hit the command Start Debugging and immediately drop into the debugger at my specified breakpoint:

 

 

Here I am stopped at my breakpoint and I am starting to step over (command o) the code.  In fact, I stepped over the code until the Python code was compiled by the IronPython engine.  So how come I could not step into the IronPython code?

 

 

If you look closely, you will see that I used the x command again to see what modules are loaded and lo and behold a module called, “debuggableSnippets2.dll” What’s in there? Using x command on the module, you can see my compiled .py code with the class name and methods that correspond to the source that I entered in the code editor.  Searching on debuggableSnippets points to this source:

 

// IronPython has two units of compilation:

    // 1. Snippets - These are small pieces of code compiled for these cases:

    //    a. Interactive console expressions and statements

    //    b. exec statement, eval() built-in function

    //    c. Types generated by NewTypeMaker for instances of Python types

    //    d. others ...

    //    All snippets are created in the snippetAssembly. These are created     using GenerateSnippet.

    // 2. Modules - Modules are compiled in one shot when they are imported.

    //    Every Python module is generated into its own assembly. These are created using GenerateModule.

    //

    // OutputGenerator manages both units of compilation.

 

Makes sense.  It’s nice that I can step through my C# code that hosts the IP engine, but what I am really interested in is stepping through the Python code.

 

I see that others have been able to do it and others here as well.  But not sure exactly “how” they are doing it.  Maybe someone will enlighten me.

 

Note in the forum that Mike Stall indicates that there may be some issues with MDbg and that perhaps an updated version is forthcoming.  That was over 6 months ago.  I really hope the MSFT DLR/IronPython teams keep tool support in mind and provide an updated version of MDgb.  Maybe Michael Foord, Douglas Blank, Ben Hall, and Stefan Dobrev can help lobby Harry Pierson to keep MDbg current and able to work with the DLR. 

 

We are all building DLR IDE’s of various sorts (IDE Developer and DLR IDE on Codeplex) and MDbg seems the most likely candidate as a debugger – for those that want debugger support.  Also, I suspect that this is a precursor for people wanting to create their own languages on top of the DLR, so you would think debugging tool support would be important.

 

In the meantime, if I can only figure our how to get MDgb to debug IronPython code…  And I may have to wait for Beta 4 to try it out on IronPython 2.0.

I was reading an interesting post at Ted Leung’s blog called, “IDE’s and Dynamic Languages”.  It is interesting to me for a number of reasons.  One is how a Text Editor can be considered an IDE, even though Ted does say that automatic syntax verification and code completion is certainly beyond a Text Editor.  

 

One thing that did surprise me was no discussion on debuggers as part of an IDE.  How can people code using a text editor without a debugger?  I guess I have been (totally) spoiled back in the VB6 days (yes, I will admit it) that I could step though code and when I bumped into an error, I could back up the debugger a few statesments, make my correction and keep on stepping through.  I have never been as productive since!  Know what I mean?  Yes, I know this says nothing about design, but an IDE is a tool for using a programming language, yes?  So how come we (as in developers) have so few tools or is that choice of tools?

 

According to a recent analyst report, 97% of developers on the .NET framework use Visual Studio and over 70% use Eclipse or Eclipse based IDE’s for Java.  As much as I love Visual Studio, and being a .NET developer, I have no control as to what I want in an IDE.  Worse yet, with the emerging new dynamic languages, IronRuby, IronPython and Managed Jscript, have almost no tool support at all in Visual Studio.  While there have been some announcements, and articles and some tooling, its like bolt-ons to Visual Studio and still yet to come.

 

As a .NET developer, specifically a .NET web developer, I would like to use something other than Visual Studio to develop web applications using a dynamic language. My wish list is for something lightweight and web-based so that I can explore using, an interactive interpreter and a simple code editor just using a web browser.  Maybe something like this:

 

 

 

And this:

 

 

This was my first version of a web-based “very” lightweight IDE that used a JavaScript based interactive console and a JavaScript based code editor.  You can download the code from Codeplex

 

Several months have passed since I released this and I have been working on a new version that targets the IronPython 2 and the DLR.  In that time, other folks have started to develop similar web-based interactive consoles and code editors.  One example is Jim Hugunin’s DLR (interactive) Console which you can download from the Silverlight Dynamic Languages SDK.  I should point out that this is a modified version.  Also note that it uses Silverlight:

 

 

As an old ASP.NET/AJAX/JavaScript/HTML/CSS developer, I am quite excited about Silverlight for a number of reasons.  I am hoping the adoption rate and the tooling for this technology also increases over time.  Silverlight development also suffers from not being fully integrated in Visual Studio, but I should be fair as it is also Beta.  But trying to run a Silverlight “web” application on a web server and interacting with other lanaguages is tough at best.  But the rich UI experience is really quite nice compared to old school ASP.NET forms apps.  So even after 6 months of releasing my lightweight dynamic languages development environment, I am still torn between “tried and trued” JavaScript and the new Silvelight.

 

I thought about how I would implement intellisense in a JavaScript console and cringed at the thought of actually how to implement this.  I could see using a few of the controls from the AJAX Control Toolkit to implement this, but it would be quite the effort.  Not only does the DLR console support intellisense, but so does Daniel Eloff's interactive console called :

 

 

Wow, I am impressed!

 

Here is another web based shell (that you cannot download) from Oleg Tkachenko:

 

 

Also Michael Foord has a Silverlight based Python in the Browser:

 

 

Jimmy Schementi has an IronRuby console:

 

 

 

Of course there are other implementations, but they are not web-based.  Nonetheless, Joe Sox’s IronTextBox works very well:

 

 

And Stefan Dobrev's DLR Pad:

 

 

And Ben Hall has just released his IronEditor:

 

 

So what’s my point?  I think all of these projects are great and kudos to the people that built them.  It takes a lot of time and effort above and beyond just regular work hours. I have been there myself, my hats off to you folks!  But, there are 8 versions of the interactive console and a few versions of a basic code editor.  I know it may be a dream, but it would be great to collaborate with these people and write out a simple set of requirements for what a great DLR console and code editor would be.  And then as a virtual team, implement it.

 

After all, to a large degree, it will be how well supported the language is from a tools perspective that will really determine the rate of adoption.  And right now, the tools (or IDE) experience for Dynamic Languages on .NET is severely lacking to the point of having several people independently developing their own tooling.  In this post I only pointed out a handful of these tools and I know there are others, but I was really targeting web-based IDE’s.  Maybe that is an opportunity?  Or is it a pipe dream?

 

I am also wondering how, in a code editor, one could hook up a debugger to actually step through the code, regardless what Dynamic Language (or Static or JavaScript or Silverlight) is being used?  Hopefully I won’t have to wait until PDC 2008 to see what the next step is from MSFT.  Who knows maybe there is enough interest to develop a Web-based IDE for Dynamic Languages on .NET.

Official website: http://globalsystembuilder.com   Download source from CodePlex: http://www.codeplex.com/gsb

In part 1, I introduced a distributed programming environment called Global System Builder (GSB).  The IDE is an ASP.NET web application.  Another part of GSB is a Winforms application that runs on a remote computer that hosts an IronPython engine and communicates back to the web server using Windows Communication Foundation (WCF).  Hosted in the web browser is a source code editor with syntax highlighting and an interactive console – both of which communicate with the IronPython engine on the remote computer to execute Python code (and soon to be IronRuby using the DLR).

 

Also hosted in the web browser is a Virtual Earth control for locating remote computers, a Remote Desktop control to host a user session to the remote computer and a Smalltalk-like System or Class Browser for browsing classes (i.e. Types), member info, etc., that were loaded in the Winform app (i.e. GSB Service) on the remote computer.

 

 

 

One of the features I am implementing is the capability to extract code comments from any assembly that was loaded into the System Browser via the remote service (i.e. the Winforms application on the remote computer).  The idea is that as assemblies are loaded into the remote service, WCF communicates with the ASP.NET web server by passing up reflected assembly information (i.e. TypeInfo, MemberInfo, etc., using System.Reflection) that is then bound to a custom GridView control in the web browser, including code comments.

 

I implemented an ASP.NET AJAX hover menu that will display context sensitive menu items depending in which grid you are hovering over.  These menu items are helper functions (almost like macros) that will add the necessary IronPython (or IronRuby) code snippet to the console or code editor window, for example, to add an assembly references, stub out methods, insert DSL, etc.  For this post, I am discussing a way to view code comments that are part of an assembly, which when clicked in the hover menu, will display a panel with the code comment (plus parameter info, etc.) for the item hovered over, depending whether it is a type, method, prop, etc.  Think of it as a web-based version of Lutz Roeder’s cool Reflector, but in a Smalltalk like System Browser.

 

As an aside, I have one (major) design issue with the way assemblies are packaged out of Visual Studio as there is no way to include the code comments that you meticulously put in your source code files into the actual assembly itself.  It is packaged as a separate XML file.  In my mind, packaging the code comments in a separate file simply breaks the cardinal rule of encapsulation.  I wish Visual Studio at least gave you the option to embed your code comments directly into the assembly itself.

 

 

OK, so how to get at the code comments?  Searching through my vast libraries of reusable open source code on the internet, I came across this post and a nicely packaged class called XMLComments.  Only takes one line of code to implement and retrieve code comments. Perfect. Thanks Stephen Toub!

 

 

I used System.Windows.Forms.dll as the guinea pig as it is a whopping 5 megs in size and so is the companion XML code documentation file.  Everything seems to work ok, but a few observations.  It seems even in Microsoft’s own code documentation, only 50% of the +2200 Types in the assembly have a code comment.  Also, what about assemblies that don’t have the companion XML file and therefore no code comments at all?

 

There are a couple of choices, one is to extend the XMLComments class to derive some form of comments.  Ultimately, I would like to be able to build comments into the assembly (separate XML file or assembly) by using GSB’s source code editor and the necessary comments syntax to inject or attach comments to existing assemblies that don’t have code comments (loaded into the System Browser).

 

[Update August 14, 2007  I recently came across, “Documenting .NET Assemblies” by Robert Chartier (Thanks Robert!) that does a really nice and fast job of documenting assemblies that… have no documentation.  Exactly what I was looking for.  Robert has done a really nice job here, and has given me a few ideas to rework the System.Reflection code I already use to pull Assembly Names, Types, Member Info, etc.]

 

How important is this feature to the project?  Admittedly it is not a core feature, but at the same time, the point of the System Browser is to browse assemblies for reuse, or more importantly, for programming at a higher level of abstraction, so having code comments that came with the assembly or being able to attach your own code comments to the existing assembly, would be a good thing.  In my previous life in the electronics R&D industry, getting a component, like an IC, without the accompanying documentation with the package renders the component useless.  Looking at the Types list in the screenshot above, what do you think AmbientValueAttribute does?  Not sure what to make of our software development industry where source code documentation is either non-existent or in a form or place that is makes it virtually unusable.

 

Reusing assemblies to program at a higher level of abstraction is another topic and several posts in of itself.  There seems to be a lot of issues (angst?) in our industry about reusing other people’s source code or assemblies.  I do it all the time, both professionally (aka my day job) and in my little hobby here.  In fact the majority of GSB reuses several source code libraries and assemblies.  Without the many people that have taken the trouble to build the open source components that make up this application (aside from my own code), I would never be in the position I am today with a working application.

 

Anyway, I digress.  Displaying code comments is one feature of a growing list of features I collected after presenting GSB to a few potential users (i.e. dynamic programmers).  Based on the growing feature list, I still have a long way to go – not the 90% complete I alluded to in my previous post.  Also, I have decided to release GSB as open source when the DLR (with IronPython, IronRuby, Dynamic VB and Managed JScript) reaches RTM.  This is going to take a little bit of architectural refactoring (mostly the hosting namespace) to plug in the DLR engine and provide multi dynamic language support much like seen in this excellent video with John Lam and Jim Hugunin.  All good though as I have nothing but time, yes

January 30, 2008 Update - Global System Builder is available for download at: http://www.codeplex.com/gsb/

Official web site: http://globalsystembuilder.com

This is an open source software project that is highly experimental.  In fact, it might be just plain crazy.  Our industry is a buzz about Web 2.0, SOA, distributed computing, etc.  Yet, what tools do we have to support distributed programming?

 

What I mean by distributed programming is the ability to program multiple remote computers (i.e. servers) from within one IDE instance, or as I call it, a DPE – Distributed Programming Environment:

 

 

 

This ASP.NET AJAX DPE consists of the following parts labeled in red:

 

1)  An Object or Class Bowser, similar to Lutz Roeder's outstanding Reflector, except mine is laid out in the old school Smalltalk System Brower format.  Where are the assemblies IronMath and System.Windows.Forms coming from?

 

2)  This is a Remote Desktop Connection (RDC) that allows connections to multiple remote computers.  One tab per connection.

 

3)  This is a Windows Forms application (or service if you will) that runs on the remote computer(s) that allows you to add assemblies to be reflected.  It uses Windows Communication Foundation (WCF) to communicate with the web server hosting the DPE, which then displays the reflected assembly information in the browser window GridViews.  In addition, this Windows Forms application hosts IronPython (and soon the DLR) which also communicates with the web server via WCF so that IronPython code can be executed on the remote server(s).

 

4)  An IronPython source code editor which provides syntax highlighting, handling multiple source code files, etc.  Again, using WCF to communicate with the IronPython DLL that is running on the remote server(s).

 

5)   Is the result of running the IronPython code that is displayed in the editor window.

 

6)   To find remote computers, I use Virtual Earth’s map control to display remote computer locations.  Optionally, a MapPoint Web Service can be used to upload custom locations and attributes.

 

7)   Oh yeah, an interactive command line console is a must have.

 

 

 

 

The context of which remote computer I am programming is activated by which tab has been selected.  Meaning that when I click on any tab, any related context, like other dependant tabs and grids are switched too. I add distributed computers by adding tabs for each RDC session, a source code editor, and an interactive interpreter (i.e. console) per remote computer.

 

Think of the web server as a middle-tier to the WCF enabled remote services that host’s the code for reflecting assemblies and executing Python code.

 

As I mentioned earlier, I will be embedding the DLR which will add support for more dynamic languages.  Also note that the web-based console window is generic in the sense that it can be a console to a remote cmd window or PowerShell window or…  Same goes for the source code editor, its syntax highlighting also supports multiple languages.

 

Like I say, it will either be useful or just plain crazy.  That’s why I call it Global System Builder

January 30, 2008 Update - Global System Builder is available for download at: http://www.codeplex.com/gsb/

Official web site: http://globalsystembuilder.com

Previously, I discussed a web-based source code editor for IronPython, which is part of a larger application that I am working on.  Most of the editor is complete, just working out the details on handling multiple source code files.  Today we will drive IronPythons’ interactive interpreter, using a web-based console, which is also part of my application.

 

Using the console, we can whip up a simple Windows forms application in seconds:

And here is the output:

The console is based on two components, one is Sam Stephenson’s Prototype JavaScript framework and the console GUI is brought to you by Jeff Lindsay’s Joshua, which I modified to work with IronPython.

 

Running a web console over the internet presents a few challenges.  Everywhere you read, AJAX is the sh!t.  I use a ton of it in the app I am building.  A stands for Asynchronous and therefore, as some have stated, synchronous over HTTP is bad form mate.  Well, in the case of a console application, I think a synchronous call is one (only?) way to make it “really” work:

 

//synchronous call to the server passing the IP string command and returning the response  

function getFile(url, passData)

{

    if (window.XMLHttpRequest)

    {

        AJAX = new XMLHttpRequest();

    }   

    else

    {

        AJAX = new ActiveXObject("Microsoft.XMLHTTP");

    }

    if (AJAX)

    {

        AJAX.open("POST",url,false);

        AJAX.setRequestHeader("Content-type", "text/xml");

        AJAX.send(passData);

        return AJAX.responseText;

    }

    else

    {

        return false;

    }

}

 

One nit pick with the ASP.NET AJAX framework is that you can’t do synchronous.  I would hope that the framework would support this option as a) other frameworks provide it and b) I want it man!  The point of a framework is to provide options so that a variety of similar, but different business requirements can be met with “one” AJAX framework.  I am already using 3 AJAX frameworks as each one brings a unique piece of functionality that I require.  Who knows, maybe it will be more by the time I am finished.

 

If you look at the console closely, you will see one small GUI  issue.  When parsing interactive = true with Python, visually, the next line of Python code should reside beside the “…” and not on the next line beside the command prompt (i.e. >>>).  Also note that the 4 spaces indent after the def statement does not work at the moment.

 

Aside from a couple of GUI issues, and a missing blinking cursor, there is one last feature to be implemented and then the console will be complete.

 

The astute reader may notice that the Windows application is running in a Terminal Services session, which is being hosted in my web browser. Did I remote into the web server? No.  On the client computer? No.  Where is the Windows application running?

 

January 30, 2008 Update - Global System Builder is available for download at: http://www.codeplex.com/gsb/

Official web site: http://globalsystembuilder.com

 

Why web-based?  Well, the IronPython source code editor I am working on is one piece of a larger web-based, open source application that I am building and will become clear in future posts as to the reasons why it needs to be web based.  

 

In the meantime, I thought I would write about some of the available components that provide real-time syntax highlighting and other features one might want in a web-based source code editor.  After extensive research, there are 3 web-based source code editors that I would recommend.

 

The screen shot above comes from a portion of my web-application that I am building.  I am using Christophe Dolivet fantastic EditArea.  EditArea is written in JavaScript and really is quite outstanding given the number of features it has, particularly that it is free (released under LGPL license).

 

One of my main requirements is that it can work with ASP.NET AJAX (a free framework) and can be embedded in a TabControl as you see in the screen shot.  I will discuss the web-based IronPython interactive console in another post. 

 

I am really impressed with what Christophe has done, not only is EditArea fully featured, but the JavaScript source code is extremely well written using prototypes.  Further, the documentation is excellent – you would think this is commercial based source code editor.  EditArea supports multiple languages, both the software itself and the syntax highlighting it supports, for example, in my case English and Python.

 

 

 

Another real-time syntax highlighting source code editor I have come across is CodePress by Fernando M.A.d.S.  It is also written in JavaScript and covered by LGPL.  I really liked this editor as well.  While it is not as full featured as EditArea, it does a very good job.  It also supports many languages, but not Python at the moment.  I see it is on the todo list.  While I could get it to run in a bare bones ASP.NET AJAX application, I could not get it to run embedded in a tab control.  However, I think the limitation may be my JavaScript coding skills (I am a C# programmer hooked on Intellisense) and not necessarily CodePress itself.

 

 

 

Syntax Highlighter by Marijn Haverbeke is another web based, real time syntax highlighter.  It is released under a BSD like license.  It works quite well, but not as fully featured as the other two editors.  However, it is quite fast and seamless.  Dos not support Python syntax highlighting, but as in the case of CodePress, would be fairly easy to implement.  What I found really interesting was Marijn’s story on how it was designed.  Fascinating Captain!

 

Thanks to Christophe, Fernando and Marijn for creating and developing these real-time syntax highlighting source code editors.  It has saved me a lot of time not only from building these from scratch, but also proves a theory I have about software industrialization.  That is, it has been done before and I don’t need to reinvent the wheel.  Some people call this mashups, others reusable software or open source, or…  No matter what you call it, I am thankful that so many people are willing to share their hard work.

 

When will it be released?  Not sure as there still is lots of work to be done and can only work on it part time.  I wish it was my full-time job!

 

January 30, 2008 Update - Global System Builder is available for download at: http://www.codeplex.com/gsb/

Official web site: http://globalsystembuilder.com

 Defines a VSPackage that controls the interaction between Visual Studio and the IronPython compiler.
 Participates in building, deployment, and source code control.
 Includes MSBuild targets and templates.
 Offers debugging support, but no expression evaluator.
 Displays and manages project items in the Solution Explorer.
 Participates in the Open Project and New Project dialog boxes.
 Exposes project and configuration properties, both generic and specific to IronPython, such as the "main" program file.
 References web services, databases, and other resources, but not other IronPython assemblies.
 Supports automation.
 Connects the Windows Form Designer to the IronPython CodeDom.
 Press F5 to build the sample, register it in the experimental hive, and start Visual Studio from the experimental hive.

The last statement is interesting.  For anyone that has used the pioneering DSL Toolkit knows that running this solution will launch another instance of Visual Studio, and in this case, complete with the integrated IronPython project system now running.  If you click on New Project, the following dialog appears:

And sure enough, the PythonPoject is fully integrated into this instance of Visual Studio 2005.  It works just like a C# Project Type, for example.  In other words it is seamless and consistent within the Visual Studio IDE paradigm.  A real testament to Visual Studios pluggable language architecture.

 The .NET Framework Class Library and the common language runtime (CLR) serve as the foundation for all .NET-based applications. But how much do you know about any of the thousands of CLR classes in the library and where do you begin to learn about them?

 

In this article, the author uses the number of relationships with other types to determine which types are going to affect your programming most frequently and how often you'll encounter them. He programmatically surveys the CLR library using reflection to make that determination. Following that, an in-depth examination of the nine most important CLR types is provided.