Finally ported CodeMirror syntax engine to bespin

I have written before about my attempt to port the CodeMirror syntax highlight engine to bespin. I got it mostly working now (for Javascript, HTML, CSS, PHP and even all these languags mixed together as in PHP) and have submitted an initial patch for review.

The port required quite a few structural changes of the code because, on bespin the syntax engine is running in a background thread if webworkers are available (FF 3.1beta, Safari 4beta, Webkit nightly) or gears is installed, otherwise it is still running asynchronous, triggered by a Javascript setTimeout() with value 0. This all has only been possible thanks to the groundbreaking work of Malte Ubl.

So if you wanna try that patch out today, you need of course a local installation of bespin, then apply the patch and use the command-line to enable / disable the syntax engine:

set syntax2 on / set syntax2 off

And don't expect it to be not even near as fast as the default engine. It is just an initial port, caching and other tricks to make it fast are not enabled or implemented yet.

Bespin, Mercurial and keeping your patch up-todate

While hacking on bespin I experimented some of the possible ways to keep a patch in sync with tip (this is how they call "trunk" in hg land).

Just cloning the repo for developing the patch

this is the easiest, and seems quite popular. Unfortunately you are switching physical directories when you switch between clones, and in the case of bespin, where running server-instances are tied to a physical directory, it requires either a smart approach for running the instances in parallel or easily shutting down one and starting up he other. And if you are working on different unrelated patches, it can get quite messy to do manual merges when tip changes. To sum up, easy to get started with, but lots of disadvantages.

Using branches

Branches are easy to create and easy to switch between them and they have one big advantage compared to cloning: You stay on the same physical directory if you switch branches. However, the pain of manual merges when tip changes seemed to me even worse than with cloning.

Using queues extension

this is currently my preferred way. With queues extension you have always a clean tip, can develop patches, apply them, remove them, apply /remove patch-sets and easily change the order of the patches in the set. Updating tip is as easy as:

hg qrefresh # update current patch with local changes
hg qpop -a # remove all patches
hg pull
hg update
hg qpush -a # apply all patches

What I haven't figured out yet is how to use bundles with queues extensions.

Update:
By enabling rebase extension, rebasing with tip got even easier. The lines above got reduced now to:

hg qrefresh # update current patch with local changes
hg pull --rebase

Update II:
Lately I just push my local repo with applied MQ patches to my personal public repo at bitbucket, with the purpose of facilitating the pull of my patches.

More on the new syntax engine for bespin

And so here continues my journey in porting the CodeMirror parser / tokenizer to the bespin syntax engine. While at my first iteration I basically just merged the codebase and managed some initial nicely-colorized rendering, I bypassed the mechanism which actually makes CodeMirror parser / tokenizer an interesting choice for bespin: the interruptable / resumable parsing. Now this is halfway working so let me explain it in detail:
The default bespin syntax highlighter works on a per line base and is extremely fast (magnitudes faster than any approach involving real parsing) and its performance is independent from the document size. But if we wanna have real syntax analysis in real-time, there is a price to pay for it and this price is a slower syntax engine. But with some real computer science (many dissertations have been written about parsers) and a lot of tricks, the user hopefully won't perceive any loss in responsiveness.

The streaming tokenizer / parser approach

While most parsers need to read the full document to parse it then all at once, CodeMirror is more like a streaming pipeline engine: it can start at any line in the document, after processing that line, the engine spits out immediately the tokens containing the syntax coloring information and in the near future callbacks for indentation and maybe even code-completion. So how does this engine work ? The main trick is that the parser stores its state at each line, so it can resume its operation at any line. Of course with all the details, it is a bit more complicated:

First we have our document, bespin internally represents it as an array of lines of text. Then there is a StringStream wrapper, which allows us to treat the document like a stream. Then there is a stream traverser which goes forward through the stream, passing the stream content to the tokenizer, which splits the stream into tokens and the last element in the pipeline, the parser does the main job of further analyzing those tokens. For every line the parser attaches a copy of itself (or better: its internal state) to he line. So once the document is initially parsed up to a certain line Y and the the user changes something at a previous line X, the engine retrieves the stream starting at line X, grabs the parser which has been attached to line X and continues its job there.

Of course there are a lot of things which can go wrong. For example the engine loosing sync between the stream representing the actual text lines, the document itself and the stored parsers. If this happens the document needs to be fully reparsed, which is a performance disaster. And this happens right now, because a few parts are not implemented yet, but I am working on it.

I am very excited about integrating the engine with Malte Ubi's work: offloading the engine to webworkers / gears for async background syntax analysis and merging it with the upcoming thunderhead editor component, which is optimized for calling the syntax engine only when really necessary. Right now I am messing with caching (otherwise the parser would block the UI, arrghh !!!), I try to detect whether calling the syntax engine is necessary, if not I provide cached results if available.

So the next week will be very interesting, to see how these things rapidly evolve.

Porting CodeMirror syntax parser to Bespin

The first screenshot (or the second, if the images are in a row) shows the current bespin syntax highlighting engine. It applies on a per-line base regular expressions to highlight the different parts. This approach is very fast, and new language color definitions can easily be added, but the engine is not aware of the internal structure of the code the user is typing in, and therefore such an engine cannot be extended to show in realtime syntax errors and provide intelligent code indentation and completion. The engine splits code into the following elements for colorizing:

• comments
• C-style comments
• keywords
• strings
• punctuation

The second screenshot shows an early stage of a new approach I am working on, a syntax engine which uses a real tokenizer and parser (borrowed from the CodeMirror project), fully aware of the structure of the code. So far the initial coloring is fine, but to get it work continuously and smooth and with large files, there are a lot of problems and a lot of work to solve them ahead.

The second screenshot (or first, if the images are in a row) also shows that a few more Javascript code elements got identified and colored:

• atoms
• variables
• variable definitions
• local variables
• properties
• operators

I have chosen Javascript as the first language to port to this new syntax engine, because it is the most important one for me and maybe for web apps in general. But JS is a dynamic language and syntax analys is tricky. The good thing about CodeMirror is that beside of JS there exist already tokenizers / parsers for:

• CSS
• HTML mixed (with JS, CSS)
• PHP mixed (with JS, CSS, HTML)
• Django mixin (my proof-of-concept, never got officially released)
  

This new syntax engine, if I ever get it working in an acceptable way, is not supposed to replace the existing one. If desired and enabled, it will try to provide a better user experience, but will use the original syntax engine as fail back.

Bespin - experimenting with embedded preview

While a browser based text editor or web development environment has many disadvantages compared to the desktop variant, there a also some great possibilities which haven't been explored yet with bespin. Possibilities such as having direct browser DOM access for previewing and possibly manipulating content and that is not limited to text. Images (img tags and background images) can also be edited live (with a bit of canvas magic, as e.g. in pixastic) and server round trips are only required to store the edited images. Traditional editors can only dream of this and adding this possibilities to monster IDEs makes them even more complex.
So here is my proposal (including fully functional early stage prototype) for a live preview, see the screenshots above, just clicking on the eye icon at the toolbar and at the place of the editor canvas with the HML source code, a preview panel gets blended in. Unfortunately there are a few issues. Lets take a look at the style definition of the example. It has defined properties for the "body". But we don't want to change the bespin body properties. So what I am doing to prevent a visual and usability disaster is parsing (not really in the lexical sense) the HTML template, extracting styles and injecting them as filtered CSS rules into the page when rendering the preview and removing the rules when switching off the preview.

So far this is only for HTML templates. But my idea is to have a pluggable template parser system so we can expand PHP, django or other templates. While this is not easy, I have done it before (ErlyDTL, the Django template language for Erlang) and I have used a javascript template system before: dojox.dtl - dojo django rendering. So why not use dojox.dtl for bespin template rendering ? Well, first it is specific for django, second it is text and node based (for performance reasons). That is good for an app where template parts get swapped frequently, but just for preview we don't need a node based template system, at least not for now because it adds a lot of complexity.

And one more thing to consider are the edge cases in which this approach won't work. Hopefully with enough trickery the embedded preview will work good enough in most cases. The bespin toolbar and commando line will anyway prevent a full-browser preview, unless they are blended out as well, which is currently not implemented and I am not even sure whether it would be a good idea to do so.

If you wanna contribute to the technical discussion, please user the google bespin core group.

Does now web development start to move into the browser ?

The browser has always been the most common tool to consume content from the internet. It also has served to create simple text based content (CMS). But now the tools start to appear for creating more complex content like web applications direclty in the browser. Let me list what I have tried out (or just heard of) recently.

• bespin the super-cool and wicked-fast browser code editor (which since today contains a few lines of code I have contributed)
  
• 280atlas, just been announced yesterday, an amazingly awesome looking visual builder for cappccino web apps, not available yet, except of the screencast.
• Wavemaker, they have been around for a while, very enterprisy, it did not work for my out of the box when I tried a few months ago (isn't the first impression what counts most ?).
• AmyEditor. Looks like an online version of TextMate. That is good, because TextMate looks and feels great and people just love TextMate, but is bad because I believe the browser is not the right place for embedding a desktop app alike. And when resizing the browser, that app did not resize, so that was the early ending of my exploration of that otherwise probably great app.
• Coghead. Was a cloudbased visual app builder. They failed, are on the deadpool now. At least they could sell some of their IP to SAP. AFAIK, Coghead was not open source, halfway flash based and had a lot of proprietary things, even a proprietary programming language. Maybe that is not the way to go, unless you have more salesmen than developers in the team.
  

Beside of bespin these web apps all seem to look like desktop apps. I am not sure whether this is what developers will feel most comfortable with. Look at gmail, greader and other highly successful web apps. They don't look like and feel like desktop apps.

There are also cool graphics apps from aviary, for creating artwork. They are flash based and take more time to load than Pixelmator on my laptop.

bespin now dojo-powered

well, at least on my laptop, but I submitted a patch, so if the Mozilla guys like it, I hope it goes into trunk. The screenshot shows bespin running on the brand new Safari 4 beta.

There are still things to do related to the port, for example eliminating global variables outside of dojo and bespin namespace, dojo build profile and fixing the zillons of bugs I probably have introduced (but at least one I fixed, mousewheel works on Firefox as well)

Porting bespin to dojo - 90 percent done

No I try to break the number one rule in software development: the last 10 percent take 90% of the total time. The dashboard works, except the command line sometimes having hiccups. The editor lets me type in text, but anything else breaks. I also replaced all third party libraries (in the external folder) with dojo equivalents. Here my experience in regard to prototype:

• It's an absolutely fantastic framework if you wanna simple things get done quickly and you don't care about JS design philosophies (name spaces, native object extension).
• There are a lot of utility functions for string, array and enumerable manipulation, but some of these functions are "opinionated" as all the great things from the 37systems guys. In dojo you end up cooking together your own one-line solutions for the simple stuff. It is additional work, but results in a customized and optimized solution.
• The huge amount of string utilities suggests that prototype is targeted to developers who prefer not to deal too much with regular expressions.

So what else is missing once the bug fixing is done:

• Name space cleanup: I used a all-lowercase namespace (e.g.: "bespin.editor"), because that is how dojo name space and file hierarchy looks like. I regret that, because it violates the bespin coding guideline and worst of all, requires a lot of slave work to update the documentation, so I will refactor to a name space mostly identical to the current one from bespin trunk (e.g.: "Bespin.Editor")
• There are still some global variables (in the startup scripts), but before I even suggest how to change that I need some feedback from the bespin architects.
• Testing (I introduced lots of bugs ...)
• Build script to create minimized JS/CSS for production, maybe as paver task.

Update: I found a cheat sheet for conversion between dojo <->prototype

Exploring bespin - and porting it to dojo

After some months absent from blogging and open source software involvement, the launch of bespin motivated me to get back to fun-hacking and blogging. Bespin is a browser based code editor which feels and smells like a real editor (vim, emacs, textmate), and not like a form on a webpage as all of the other browser based code editors (including an experimental one I hacked together myself). Internally bespin uses canvas and therefore currently only runs well on Firefox or webkit nightly. As JS lib Protoptype was choosen, but that might change in favor of dojo. So I thought I will give it a try, to port bespin to dojo and learn about the internals of bespin. And it might serve as suggestion in case bespin officially gets ported to dojo.

So where we are so far ? The screenshot at right shows that dashboard canvas rendering halfway works already, but with scaling dimensions something went wrong. Prototype makes the simple things very simple, dojo doesn't care much about the simple things, so I had to hand code a few things which had ready-to-use counterparts in prototype. But the really ugly thing about Prototype based javascript development is the global name space pollution and the extension of native objects, that is so much better handled with dojo and its class system. Of course nobody cares about those details once an app is up and running. Will finish the port and submit it to bespin bugzilla, but it might take a few days to iron out the bugs.

Javascript performance optimization with jeene

Karl Krukov is working on Jeene, a Javascript library for partial evaluation, based on Douglas Crockford’s Pratt JS parser. Karl has just published some benchmarks (across all common browsers), where Jeene brings performance optimization of approximately 50%.

I want chrome !

But there is no date yet when it becomes available for Mac.

Online publishing with Issuu.com

Issuu has an interesting solution for online publishing of print magazines and other publications. It is partially in flash, I usually don't like Flash, but their Flash stuff is really well done. And they take interesting approaches for the surrounding stuff of a web app as well, take a look at their documentation: http://platform.issuu.com, it's all google groups pages !

Perl on Google App Engine

Brad Fitzpatrick, the guy who added memchached to the Google App Engine, is starting a new project at Google: adding Perl support to the App Engine.

10gen: Cloud hosting platform offering serverside Javascript

There was some speculation that Google will offer it once they open up the App Engine Platform to more languages, there were some small-scale attempts with serverside Javascript (including the mine, with http://erlyjs.googlecode.com, which is currently receiving very little attention from my side) but now 10gen (with well known Java hacker and director of the Apache Software Foundation Geir Magnusson as CoFounder) plans to offer cloud-based hosting solution with any language running on the Java VM (at least that is what I am reading between the lines of their press release) and the funny thing is they don't start with Java itself, no they offer as first Language Javascript. The guys at 10gen also seem to like django, a quick look at one of the tutorials reveals they are providing a thing called djang10, django inspired framework for creating web applications purely in Javascript.

Erlang Exchange 2008

The last two days I have spent at the Erlang Exchange Conference. It was great to met other Erlang hackers in person I just knew from their blog, emails or mailing list. Below the slides from my talk:

Javascript syntax highligting for Firebug

The syntax highligting capabilities of CodeMirror, my favorite Javascript based code editor (for Web IDE-type applications) found their way into Firebug, currently available as additional extension.

Google App Engine limitations and workarounds

One of the current limitations of the Google App Engine is the lack of scheduled background processes. But early adapters are creative to circumnavigate this and other limitations. The most simple solution to this problem has William Vambenepe: he uses Google Reader to subscribe to a dummy RSS feed on his App Engine hosted test app, that results in some kind of regular polling with an interval of about 30 minutes. Of course this can't be used for any serious stuff, and the eight seconds maximum lifespan of a request also is not helping much to improve the situation. William concludes:

In the meantime, this was a fun exploration of the GAE environment. It makes it clear to me that this environment is still a toy. But a very interesting and promising one.

A more sophisticated but also much more complicated solution to the same problem has Peter Dolan: HTTPMR, a Map Reduce implementation in Python for running on HTTP based web clusters such as Google App Engine.

I am experimenting myself with a completely rewritten, not yet released version of ErlyComet, for adding a RESTful, highly scalable Comet layer with transactional cache in front of Google App Engine (or any other service which provides easy, cheap, reliable and scalable solution for persistence and shortlived HTTP requests)

Malicious Flash fullscreen mode

Go there an try it yourself. It doesn't do any real harm, but it can scare people, especially those not really familiar with the latest capabilities of the flash player, such as full screen mode. While this example is just fun and actually the first example of this kind I have seen, I guess the same principles could also be used for real malicious stuff (e.g.: a new kind of phishing).

IETester - makes web page testing on IE less painful

Previously I was switching between several virtually installed Windows XP instances on Parallels (for Mac OS X), just for testing a web page on different versions of Internet Explorer. Now I have installed IETester, a free Windows tool, which allows to run simultaneously (and switch between) the following IE versions:

• IE 5.5
• IE 6
  
• IE 7
• IE 8 beta 1

Notapad - the start of a dojo based WebIDE

Niccola Rizzo has been working for a couple of months now on the CodeTextArea dojo widget for in-browser Javascript code editing and he has just launched a public demo and test site named notaPad.

The editor provides Javascript syntax highlighting and additional functionalities with the following key bindings:

CTRL + space - autocomplete
CTRL + l - GOTO LINE
CTRL + b - GOTO MATCHING BRACKET
CTRL + c - COPY
CTRL + x - CUT
CTRL + v - PASTE
right click on the row numbers - BOOKMARKS