Erlang on Javascript

While I am working on a Javascript to Erlang compiler for running Javascript code on the Erlang virtual machine, others do the opposite: Alex Graveley has written Er.js, a Javascript library for using Erlang-style concurrency with JavaScript, even with an Erlang like syntax !

Javascript does not support natively any kind of threads, therefore a lot of tricks (like using the asynchronous timer callback) went into this and other narrative Javascript based threading libraries to make it look and feel like real multithreading.

Gmail insecurity - check your filters - you may have been hacked

David Dairey's .com domain name got stolen by an online criminal, exploiting a gmail security vulnerability (now fixed). The scary thing about this is that the exploit added a filter to gmail, so even after Google fixed the security whole, any malicious inserted filter continues to do its evil actions. Read the entire story and check your filters !

How to put HD video on the web with latest flashplayer

First I want to make it clear that I am all for open web standards and requiring the latest flashplayer for showing some video on your site always degrades the user experience, no matter how much effort you put into avoiding that. So if you wanna depend on latest flashplayer, there must be a strong reason for it. Flashplayer 9.0.125 (released about a week ago) offering high quality codecs (h264/HE-AAC) might be such as strong reason, the audio/video quality is excellent and filesize of the videofiles is relatively small. Here the required steps to produce such a video and how to put it on a webpage without degrading the user experience too much.

Let's assume you already have your original video file in high quality and large file size, either form a HD video camera, a screen recording application or most probably as a result of post processing with a video editing software.

First you must transcode your media to h264 video / HE-AAC audio and put it in a mp4 fileformat. I assume there exists some software from Adobe which does that for you, but I am not familiar with their current commercial offerings. In my case I put together an open source toolchain to perform the transcoding.

Now you have the video as a *.mp4 file. Next you need to prepare your website to embed the video with a flash videoplayer (there are others which work equally well, just the code below probably neds to be slightly modified). The tricky part is that you require the latest flashplayer, so I suggest embedding via Javascript and with SwfObject, that allows to upgrade to the latest flashplayer via Adobe ExpressInstall, if Javascript is enabled. And what if Javascript is not enabled or not available at all ? There is a solution for that: "Extended" markup, which results in either the video or an alternative content (bur no easy upgrade via flashplayer ExpressInstall). Here the embedding steps in detail:

Load swfobject.js (Javascript library):

<script type="text/javascript" src="{{ path }}/swfobject.js"></script>

Register the video (via Javascript):

<script type="text/javascript">
 swfobject.registerObject("{{ video-DOM-ID }}", "9.0.115", "{{ path
}}/expressInstall.swf");
</script>

Extended HTML markup for embedding video:

<object id="{{ video-DOM-ID }}"
classid="clsid:D27CDB6E-AE6D-11cf-96B8-444553540000" width="800"
height="620">
 <param name="movie" value="{{ path }}/mediaplayer.swf">
 <param name="allowfullscreen" value="true">
 <param name="menu" value="false">
 <param name="flashvars" value="file={{ path_to_video }}&image={{
path_to_preview_image}}">
 <!--[if !IE]>-->
 <object type="application/x-shockwave-flash" data="{{ path
}}/mediaplayer.swf" width="800" height="620">
 <param name="allowfullscreen" value="true">
 <param name="menu" value="false">
 <param name="flashvars" value="file={{ path_to_video }}&image={{
path_to_preview_image}}">
 <!--<![endif]-->
 <h2>To view the video:</h2>
 <p>
   <a href="http://www.adobe.com/go/getflashplayer">
           <img src="http://www.adobe.com/images/shared/download_buttons/get_flash_player.gif"
alt="Get Adobe Flash player">
         </a>
 </p>
 <!--[if !IE]>-->
 </object>
 <!--<![endif]-->
</object>

Remarks:

• all {{ path }} variables need to be replaced with the actual path or URL where you have your static content.
• {{ path_to_video }} : the path or URL of the actual video file
• {{ path_to_preview_image }} : path or URL of a preview image

Example:

• as explained above, and inside this blog (requires editing the blogger template, if you wanna try that out yourself)

ErlyComet article and screencast

Today my article "Getting started with Comet on Erlang" got published at CometDaily, there is also a screencast there (scroll down there, to the ErlyComet section) if you prefer to watch and listen ...

Update: Some users could not load the screencast / did not like to get asked for upgrading the flashplayer. I halfway expected this. It's not that I haven't put in efforts to avoid that kind of bad user experience, but my efforts apparently failed. Let me explain in detail: the video is encoded in h264/HE-AAC and that only works on flashplayer 9.0.125. If the flashplayer is not the very latest one, a dialog should inform the user (and provide direct link) to upgrade the flashplayer, if he wishes to watch the video. Uploading the original QuicktTime screencast to youtube or google video results in bad, mostly unreadable text quality, similar to all those RubyOnRails screencasts which have been originally recorded in QuickTime.
Nevertheless, here is the google video version (bad quality).

The web hosting services of different hosting companies include the most additional features of domain registration free of charge. The web site design of varied websites is developed by the professional web designers who have artistic touch of Excellency. If you want to have direct and fast internet access, you should subscribe reliable broadband service providers. The website development services are offered by the different IT companies to boost up the significance of web site into the main search engine.

Erlang TextMate integration - Documentation lookup

I have made some progress with ErlyMate, my pet project for adding first class Erlang support to TextMate. Based on a new architecture with an Erlang background server (starts and stops together with TextMate, wrote a Cocoa plugin for that) I implemented a very first and highly experimental approach of edoc lookup. More technical details can be found at the googlecode project page.

And here a screencast to show you how it looks and feels like (requires latest flashplayer, and asks you to upgrade to it, if you don't have it, because video is h264/HE-AAC encoded):

Rubinius - the next generation Ruby virtual machine

The guys at Engine Yard, a RubyOnRails application hosting company are serious about solving the Ruby performance issues (Ruby is interpreted) and they are developing a virtual machine. Straight from the Rubinius homepage:

Rubinius draws on the best virtual machine research and technology of the past 30 years and incorporates the newest research in dynamic language implementations. Rubinius implements the core libraries in Ruby, providing a system that is much more accessible, easier to develop and to extend.

And what's in the bag for an Ex-Ruby-developer (Ruby it is a great language, but since I am comfortable with Erlang, I don't have any use for Ruby anymore ...) ? Well, TextMate, my preferred editor has a lot of its functionality implemented as Ruby scripts, and if they run faster with Rubinius, that's fine for me.

Benchmark - ErlyWeb vs. RubyOnRails

Yariv Sadan, the author of ErlyWeb, has benchmarked his Erlang MVC web framework against RubyOnRails 2.0 and according to his results, ErlyWeb is outperforming RubyOnRails by factor 47. [Corrected: it's 6 now, see Update]

If Ruby is on Rails, than ErlyWeb is on rocket engines !

Update: This number is far too good to be true ! In fact, Yariv had Mongrel running in non-production mode, in his corrected version the ErlyWeb outperforming factor went down to 6.

Erlang R12B-0 on Leopard

Yesterday Erlang R12B-0 was released. Compilation on Leopard (with XCode development tools installed) was straight forward, except for one Erlang application: the new profiling tool percept, which got disabled because of missing libgd library at the ./configure step.

So I installed it via macports:

sudo port install gd2

unfortunately ./configure continued to complain. Finally today a solution for how to run ./configure was posted on the Erlang mailing list:

./configure --with-gd=/opt/local

and everything went without further problems ...

There a lots of new features and improvements, and I look forward to explore them one by one in the next weeks and months. Just the troublemaker percept I had to inspect immediately. It seems to be a powerful tool for performance analysis and it ships with a module called edg which wraps libgd and serves for creating 2D vector graphics (to visualize the percept profiling results), similar to my own erlycairo, but it produces*.gif files (erlycairo only outputs *.png files).

Updated ErlyCairo

Did some housekeeping on my growing inventory of Erlang open source projects targeted to web development. I completely rewrote the Erlang part of ErlyCairo, the Erlang bindings for the Cairo 2D graphics library. Previously it had a hard coded node name for the C-node, now it is based on a much more flexible gen-server approach.

Below see how to use this software to create a simple, purple, 100px * 100px PNG image:

start() -> 
 erlycairo:start_link().

%% if you run multiple C-Nodes, to avoid nodename conflicts 
start(CNodeNumber) -> 
  erlycairo:start_link(CNodeNumber).
    
stop() -> 
  erlycairo:stop().        

create_images()->
  rect("rect.png", 100, 100, {1.0, 0.2, 0.7, 1.0}).

rect(File, Width, Height, {Red, Green, Blue, Alpha}) ->
  case erlycairo:new_image_blank(Width, Height) of
      ok ->
          erlycairo:set_source_rgba(Red, Green, Blue, Alpha),
          erlycairo:rectangle(0, 0, Width, Height),
          erlycairo:fill(),
          erlycairo:write_to_png(File),
          erlycairo:close_image(),
          ok;
      {error, Reason} ->
          exit(Reason)
  end.

Amazon EC2 / S3 Network Performance

Thorsten von Eicken (CTO at RightScale) did some bandwidth measurements, here an overview of his analysis:

EC2 <-> EC2 (large instances):

• one connection: totally 75 MB/s
• three connections: totally 96 MB/s

S3 <- EC2 (large instance, HTTPS, download):

• one connection: totally 12.6 MB/s
• eight connections: totally 49.8 MB/s

the results for non-SSL (HTTP) are about the same.

S3 -> EC2 (large instance, HTTPS, upload):

• one connection: totally 6.9 MB/s
• twelve connections: totally 53.8 MB/s

TextMate is the most popular Rails development environment

So I am hopefully on the right track with my attempt to improve Erlang support for TextMate !
Tim Bray asked 1000 Ruby / RubyOnRails about their preferred development environment, below just the most popular ones, for the complete table go to Tim Bray's Ruby survey.

	All	%	Ruby	%	Rails	%
TextMate	478	30.35	216	24.24	262	38.30
Vi family	345	21.90	217	24.35	128	18.71
NetBeans	184	11.68	93	10.44	91	13.30
Eclipse	175	11.11	101	11.34	74	10.82
Emacs family	198	12.57	132	14.81	66	9.65

Disaster recovery nightmare with mozy.com

Shit happens and usually when you don't expect it, in my case I lost some personal data on my mac (iTunes and iPhoto library). I had signed up with Mozy for automatically backing up my data and easily restoring it, so I thought it just needs a few mouse clicks and everything is fine again. Unfortunately that was not the case. The pictures above show how both restore methods they offer (online restore and download-file-and-restore) failed (even after multiple retries).

Mozy support staff seems to responsive and friendly, but that's all. The copy-pasted text below is from an email they sent me 21 hours ago:

I will continue to do more research on this, and tomorrow our mac specialist will be in the building and from there we can work on a more permanent solution.

Lessons learned:

• Don't trust your data to a fast growing startup company. In my case it was Mozy. But I believe this can happen with any other of these over-hyped cheap-service companies.
• Don't test the backup service by just restoring a simple file. Take the time to do a test restore of your real valuable data at its full size.
• Choose at least two totally independent backup solutions.

Faster interaction between Erlang and TextMate

My first attempt of combining TextMate key bindings to Erlang scripts was fully based on escripts. Unfortunatley starting an escript costs some time (because of the startup of an Erlang VM), on my mac it is about 0.15 seconds. Not much, but enough to be perceived as short delay by the user. So I was looking for a faster way of interaction and Erlang core developer Bengt Kleberg pointed me to erlaunch, a kind of erlang scripting environment where the VM runs in the background and dosent need to be restarted for each script. So I guess something along that lines is the way to go for TextMate Erlang integration, probably a TextMate plugin written in C which takes care of the Erlang background VM during TextMate uptime. Need to learn now how to write a TextMate plugin ...

s3fox - a great Amazon S3 explorer and backup tool

I recently had to do some disaster recovery on my mac and made the painful experience that mozy sucks in every way one can imagine. After several failed attempts I managed to restore most of my stuff. Ok, mozy mac version is still in beta, and it is dirty cheap but it is also completly useless for the less experienced user, because it was not straight forward for me to restore large sets of files and it is useless for the more technical user, because it gives you little control of what it is doing and you can't even backup hidden files (e.g. a git repository) and when I asked them about that, I just received a non-answer from somebody who did not understand what I asked. This is my personal experience. I am sure there are plenty of happy mozy users out there.
So I had to reevaluate my options for personal backups. Because I was already a heavy user of Amazon S3 in relation with various web projects, I tried out some of the common S3 desktop tools:

• JungleDisk (commercial, cheap) - sucks because what you store with JungleDisk you can only retrieve with JungleDisk (with additional effort you can do anything of course).
  
• S3 Browser (free) - sucks because there is only a mac version and it has very limited functionality.
• BucketExplorer (commercial, free while in beta) - sucks because it is written in Java and therefore looks ugly and just feels strange on the mac.
  

It's a crowded market and there are a lot of other S3 explorer and backup tools out there. I actually had been using one heavily and with great success when it came out: s3fox. It's a slick Firefox extension and it was the only one which exactly worked the way I actually expected it to work. But there was one problem: after initial release, the author did not maintain it, and Firefox evolved, the S3 protocol itself evolved and when I switched to a mac half a year ago, I couldn't use it anymore (I don't remember for what particular reason). Now I checked again and - nice surprise - there is a new version (0.4), which solved all the problems and even got drag & drop and some functionality for synchronizing local and remote folders. Easy to install, easy to use, free. I didn't have to look any further.

MochiWeb got a HTML parser

Since MochiWeb went open source I have been working with Tait Larson on a Comet server (HTTP push) built on top of MochiWeb. We are not there yet, but it has been a very pleasant experience so far, especially considering an earlier (failed) attempt of mine to build such a thing on top of yaws, where I had to patch yaws and deal with all kind of annoyances (don't get me wrong, yaws is great for 99% of all possible use cases, just HTTP Push belongs to the other 1%).

So today I was just doing some online housekeeping and I noticed that MochiWeb got a HTML parser. Thats great ! So far, it has been asked many times on the Erlang mailing list how to parse HTML. And sooner or later somebody points to the yaws HTML parser, which works reasonably well. One time it was me asking that question and when I got the answer, I started to play with that yaws HTML parser and some simple XHTML (if I remember properly) examples and everything looked fine. But things turned nasty when I tested the yaws parser with real world HTML.

Now I hope people are starting testing and crashing (I just did) the MochiWeb parser with real world HTML and provide feedback to the developers so they can further improve it !

Rounded corners with canvas tag graphics

There exist many different approaches for creating rounded corners on HTML pages, and each of these approach has its problems. Javascript based solutions consume too much CPU cycles (and even more if doing anti aliasing), background-image based solutions are complicated (and even more, if done properly, e.g. sliding-doors based) and involve additional HTTP requests and solutions based on a bunch of DIVs (with border and varying margin to emulate the corner radius) turn any page into a horrible tag soup. The slickest solution, with the cleanest HTML and Javascript code I have seen so far is Greg Houston's mocha. It uses the canvas HTML tag, which is supported by Firefox, Safari and Opera 9. The ugly part of this solution is the heavy work-around required for Internet Explorer.

Amazing browser-based Javascript IDE

A few months back I put together a summary about browser-based Javascript code editors. Then I tried myself to create one, with dojo, but I did not get very far. Then Nicola Rizzo released CodeTextArea, which is a dojo widget and a very promising approach. A demo is available and it currently best works with Firefox on MS Windows. Today I learned about TIDE, a full fledged browser-based Javascript IDE, which loads fast, looks amazingly good, provides a lot of typical IDE functionality and has even a bunch of working Javascript demos to play with. It uses the following open source libraries:

• ExtJS for the GUI
• CodePress for the Javascript Editor
• Narcissus (from Brendan Eich) for Javascript parsing

The webhosting reviews of the different hosting companies give very informative and useful details about their hosting packages. The basic aim of seo services is to provide all essential tools of search engine optimization to enhance the web ranking according to the criteria of main search engines. The domain registration service providers offer very affordable rates for registering domains. The one of most leading hosting service providers, dotster has been offering very reliable hosting packages for the potential clients in reasonable ratings.

Amazon S3 will soon support upload via HTTP POST

One limitation of Amazon S3 is its strict REST behavior: upload only via HTTP PUT, and that is not possible with most of the browsers. Today a proposal was published by Amazon staff, so the final solution is probably not too far away anymore.

Easy Erlang compiling with TextMate

Today I created a custom command for easy Erlang compiling with TextMate. Errors (if there are any) are popping up hyper-linked on a HTML window, and if you click on them, you get straight to the file and line with that error.

• Download: Erlang Build command
• See it in action: Screencast (Quicktime, ca. 4 MB)
  

This is work-in-progress and just compiles the current file. The following features will be added in future versions:

- only compile modified files
- parse Emakefile if available, otherwise:
- crawl directories to figure out src, include and ebin folders.
- code reload if node is running

Which Javascript library ?

Update: Simon Willison's Javascript library slides, which where embedded at this place, are not accessible any more, so I took out the embedded slide show.

If it is just about pepping up a webpage with some AJAX and some flash-imitating eye-candy, any of the popular Javascript will serve similarly well. But when it comes to full fledged web applications, the dojo toolkit is leading the pack. Vance Dubberly shares the same vision and today he has provided a detailed answer on the dojo mailing list, why he is choosing dojo:

Because it's an application development framework. Most of the
javascript libraries out there serve some special purpose, Ext is a
gui toolkit, Prototype is a util library, and everything else is
pretty much the same. Dojo holds a special place in the community in
that it provides a unified framework for developing applications in
javascript. What I mean by this is that it provides everything from
simple utility classes to full implementations of design patterns.
Dojo helps me organize my application layout, debug it's output, and
prepare it for distribution.

Drawbacks? The most glaring one is the horrible documentation. It's
gotten a lot better but it's still neither consistent, clear, nor
complete. The documenters are working mostly API style docs. The dojo
book reads like a brick wall. The project is badly in need of some
"Here is how to accomplish (X) style docs" Despite this the pay off is
high for those who invest the time and suffer through the exponetial
increase in 4 letter words coming from between their lips.

The widgets are a nice bonus but I gotta tell you I used it before .9
and wouldn't touch the widget system because it was heavy, slow, and
well, annoying. ( no longer an issue ) It has much to offer beyond
widgets. Take a look through the dojox directory ...

Large file sizes aren't really an issue anymore, and what's better is
that you can if you choose, optimize your builds to include only what
you want in the dojo.js file.

Oh and one last thing, a thing which I think is often overlooked. The
Dojo foundation isn't simply building a javascript library. They spur
and drive innovation in the community often developing or incubating
break through technology. This kind of behavior deserves to be
rewarded.

First step in adapting Textmate for Erlang

I have been using Textmate for two weeks now, for all my coding needs, my productivity has increased (I know, all the real good programmers use Emacs or Vim, so I am not one of them). Today I spent some time on starting a little side project: improving Textmate Erlang integration. I have created templates with skeletons for a general Library module and also for the the following OTP behaviors:

• application
• gen_server
• gen_event
• gen_fsm
• supervisor
• supervisor_bridge

The module header details such as author, license, etc. can be specified via environment variables.
There is a long way to go, to create a complete Textmate bundle and reaching an Erlang integration as provided by Emacs. If anybody wants to join, you are welcome. An feel free to use the templates and adapt them to your own needs:

Download: Textmate Erlang templates

Note: if you set environment variable TM_MY_LICENSE to "MIT", then the MIT license gets included in the header as comment (more open source licenses to be added). Other custom environment variables I use: TM_MY_EMAIL and TM_MY_WEBSITE. You can set them in the preferences.

Mochiweb Erlang HTTP server toolkit now on googlecode

Mochiweb is a small and extensible web server written in Erlang. I have mentioned it before. Today Bob Ippolito and his team at Mochiads have started the mochiweb project on googlecode:

• mochiweb.googlecode.com
• groups.google.com/group/mochiweb
  

I already have hacked the sources a bit, just to make it better fit my needs (e.g.: adding fd_server) but the real fun begins when people start to extend it. I mean extensions which do not modify the core engine, and which can be separate projects at different repositories. Anybody interested in collaborating on OpenId, oAuth and COMET extensions ?

Open social insecurity

Everyone is praising google's open social initiative. So let me do the opposite and try to put together a few weak points:

• Open social apps already have been hacked.
  
• Open social is not really open yet, only the client API has been published so far. It allows to put iframe apps with Javascript based business logic (or should I say "social logic") into open social containers such as orkut. Currently you have to join the waiting list, to get notified when the server API specification gets published.

Dave Winer goes even a step further and criticizes the motivation behind open social:

Standards devised by one tech company whose main purpose is to undermine another tech company, usually don't work.

In this case it's Google trying to undermine Facebook.

And I don't think it's going to work.

Time will tell ...

[Update:] Just after posting, I found more critique thoughts, by Julien Bond:

As a geek it pisses me off because there's absolutely no accountability or transparency in how those standards are developed. It's every bit as bad as MS trying to force the Word XML standard through the standards bodies. Google is something of a black box. There's no way to influence them. Stuff appears out of the black box fully formed.

And here a detailed article about, exploiting open social XSS vulnerabilities on ning.

Erlang and textmate

Two days ago, I felt the need to reevaluate my options for an Erlang text editor or an IDE. I have been using aquamacs (the mac-ified emacs) for a couple of months now. It has a steep learning curve, but looks reasonably well on the mac (yes, that matters for me, and therefore I try to avoid Java desktop apps) and it provides good Erlang integration. And everything can be customized, in elisp, a LISP dialect. That is what I did not like. I would prefer to customize things in Erlang itself.
So what are the other options ? Beside of of emacs, a vim hack and Java IDEs (erlide, ErlyBird) there are no other open source options I am aware of. So I investigated about proprietary solutions and it turned there is one with decent Erlang support: Textmate. That was a pleasant experience, quickly downloaded and installed and it looks great. Textmate ships without Erlang support, but it's very easy to install the Erlang bundle, I just had to install and run GetBundle, a helper app to browse and install additional language bundles.
The support for Erlang is not that much yet:

• Syntax highlighting
• Code snippets

There is also a language bundle for Makefiles, but it seems to simplify only running make without arguments, therefore it is not very useful. The most interesting part, which I haven't explored yet are custom scripts accessible via user-defined key-binding, which can take current file or text selection as input, process it and overwrite the input with the result. Those scripts can be in any scripting language, so it should work with with Erlang escripts. Will experiment with this ...
There exists also a windows editor called E-TextEditor, which is language bundle compatible
with TextMate.

The Case for the Open Web

Alex Russel, dojo toolkit project leader, presented recently the slides below and I spotted "Web 3.0" on the cover slide ...

Prism - Turn any web page into a desktop app

Today Mozilla Labs launched Prism, a tool for easily turning web pages into desktop apps with their own window, desktop shortcut but no URL address bar (everything customizable, of course). It's not a new project, just the relaunch of what was previously mostly unknown as WebRunner. If the Mozilla guys do it right this time, Prism will compete head-to-head with Adobe AIR.

An early alpha version, currently only for windows, is available for download. I tried it out, for this blog, it is very easy to use, just one dialog for defining Name, URL and optional properties for the web page to be turned into a desktop app.

Google AdSense might mess up your page

Javascript expert Andrea Giammarchi points to some potential flaw in Google Adense. If you put on the same page Google AdSense and your own or other third party Javascripts, AdSense could mess with the others in a destructive way so that nothing works anymore on that page. The origin of the problem is that AdSense deletes properties of global objects, without checking whether these properties actually belong to AdSense.

What can one do to prevent problems ? The usual recommendation for dealing with the insecure aspects of Javascript: don't use global variables and put your stuff in your own namespace. See also my previous posts about Javascript security.

HTML5 movie trailer

Mixing HTTP and HTTPS without getting browser warnings

Great tip by Ned Batchelder: On HTTPS sites you can link to HTTP assets without getting a browser warning by using instead of

  http://fast.cdn.net/pix/smiley.jpg

the following syntax:

  //fast.cdn.net/pix/smiley.jpg

Why Erlang ?

Other functional languages such as Haskell or OCaml are for most people just a bit too academic. RubyOnRails is reaching end-of-hype, it's underlying language Ruby is dog slow and not scalable. Java is far too complicated for me and without blooooated tools such as eclipse or IDEA, it is a just mess to deal with, at least that was my painful experience I made recently: I developed a prototype of a Flash video streaming server in Erlang. There exists a more complete Java open source equivalent: Red5. I have a couple of years of experience as Java developer, so I thought the easiest was just to analyze the Java source code and port it to Erlang. Wrong. The Java code is so complex, that is was easier for me to reverse engineer the proprietary RTMP video streaming protocol by analyzing the TCP/IP packet flow of a running Red5 instance by using a network sniffer instead of reading its source code. Is there any other language I think one should not choose instead of Erlang ? Python. I actually had only heard good things about it, until I got to know about its dictated indentation style, which put an abrupt end for me to any further digging into that otherwise probably great language ...

Ok, end of ranting. Developers are a highly opinionated species. Sometimes discussions among developers about the right programming language take amusant, ridiculous and even religious dimensions, despite everybody trying to be objective (well, expect for his/her preferred language). So just use the right tool for the right job and don't listen to me... But listen to what the really smart and experienced people have to say. There is an interesting thread currently going on at the Erlang mailing list. A Java coder asks whether he should learn Erlang or OCaml next. Bob Ippolito (recently interviewed) says:

When I was evaluating Python alternatives for building the core
technology behind MochiAds I tried out a bunch of languages and Erlang
was the only one that was easy for me to learn and had the right
balance of features, performance, and reliability. A year later we
have about 16 machines running 80 Erlang nodes powering about 16
different "components" of our infrastructure and 4 people working on
it at the moment (originally it was just me). It worked out so well
that we rewrote the server component of our MochiBot service in Erlang
and we've been using it to build lots of internal tools such as our
monitoring software, our single sign-on service, etc. as well. None of
us had previous Erlang experience, but we're all very comfortable with
it now.

After about a year with Erlang, I'm not sure I could part with hot
code loading, light-weight processes, and multiplexed socket IO for
writing servers. Also, Mnesia has been really useful to us to
temporarily store "real-time" data (ram_copies) so that we don't have
to make users wait for it to get batched into the SQL databases. The
distribution stuff mostly Just Works once you figure out how to set it
up (though we did have one bad experience with a network partition due
to a switch acting up, it was recoverable manually).

O'Caml is a useful language too, but for writing a network app I can't
really imagine going with anything but Erlang if you're looking for
redundancy and scale. Unless you want to write your own half-baked
Erlang-like system before even trying to solve something a little
closer to your actual problem domain.

Joe Armstrong, one of the original authors of the Erlang language and Programming Erlang author says:

Erlang/OCaml/Haskell belong to the same language family - if you learn any one
of them then learning the next one in the family will be a lot easier
than starting from scratch.

These language differ - but have the same core concepts - the idea of immutable
state - programming with immutable state is the thing that you need to learn.
The details of how you do this vary from language to language (you can
use processes
with tail recursion to model state in Erlang, or monads in haskell, etc.).

I'd start with the language that most suits your problem domain - a
rough guess might
be to think of these languages as follows:

OCaml - use as a replacement for C - good for implementing virtual
machine emulators
tightly coded non-distributed applications.

Erlang - use as a replacement for Java - good for programming
distributed fault-tolerant
applications - good support for multicores/concurrency. Good as a glue
language to
glue together components co-ordinate activities on different machines etc.

Haskell - use for implementing domain specific languages, symbolic
computations etc.

And what has Erlang in the bag for Web developers ?

Not so much yet, if you look for an easy-to-learn, convention-over-configuration one-size-fits-all framework a la RubyOnRails. Web companies use Erlang today to overcome scalability problems for web based instant messaging (e.g.: ejabberd at twitter and Meebo). Among the few publicly known partially-to-mostly Erlang powered sites are MochiAds and Slideshare.

Here a little overview about some Erlang based web servers and frameworks:

• Yaws, the most popular Erlang web server. Active development since 2002. Many contributors, lots of add-ons. If you look for an Erlang-based, Apache-like web server, than yaws is the right thing for you.
  
• Erlyweb, by Yariv Sadan. The most popular Erlang MVC framework, built on top of yaws, enables you to easily do any-web-thing you can imagine, if you are a comfortable with Erlang and don' t mind to integrate yourself the AJAX toolkit of your choice. But don't expect a learning curve as with RubyOnRails, where you can start in the morning, without ever having heard anything about Ruby before, and at night you have your first simple web app running and have learnt Ruby without even noticing it.
  
• Tercio, by Eric Merrit. Different philosophy than Erlyweb, targeting AJAX apps which do most or even all rendering at client side.
  
• Mochiweb by Bob Ippolito. My preferred toolkit to easily build a custom HTTP server.
  
• And last and least, a shameless plug for my own upcoming web framework and service, which aims to lower the barrier to entry for individuals and companies doing utility computing based development and hosting of scalable AJAX / Comet web apps. You won't even need to know Erlang to start with, unless you want to customize the framework itself. More about this when I actually have something to show ...
  

Amazon EC2 instances now with up to 15 GB RAM

Utility computing is getting more and more interesting for scalable webhosting. So far Amazon EC2 only had one instance type with 1.7 GB RAM, 160 GB HD, 32-bit platform and $0.10 per instance hour. Now there are two new Amazon EC2 instance type, both 64-bit platform and with significantly more power:

Large instance (new):
7.5 GB RAM, 850 GB HD, four times more computing performance, $0.40 per instance hour

Extra large instance (new):
15 GB RAM, 1690 GB HD, eight times more computing performance, $0.80 per instance hour

I am working right now on some Erlang tools to simplify web application hosting on utility computing infrastructure and look forward to get my hands dirty on this new EC2 instances !

Improving web application security without degrading user experience

Generally speaking, web application security involves a trade-off between security and the user's convenience. But I am not generally speaking in this article. I am trying to explore some areas where it is possible to improve security and let equal or even improve the user experience.

Single-Sign-On with OpenID based authentication

First, this makes live easier for the application developer, because to start with he has only to implement a client library (available for PHP, Ruby, Python, ...) at his server back end and not a whole authentication server. The end user gets a Single-Sign-On and he can chose from any public OpenId provider he wants. That's the theory. In practice, OpenID has been something new, just confusing the user. But this is changing rapidly, according to this document, the adoption rate is exponential and there are already 120 Mio OpenIdDs out there.
Because OpenID decouples the choice for an authentication server from the web application itself, the user can minimize the trade-off between security and convenience purely based on his preferences or his particular situation. And now there exist also innovative, password-less approaches:

• Image-sequence based login: Vidoop. Instead of a password, the user needs to remember an image sequence. There are quite a few interesting facts about Vidoop: e.g. it can generate ad-revenue from the images for the Web application provider, it is resistant against repetitive logins with a stolen token (the image sequence) because it is based on a challenge-response method which is different for every login attempt.
  
• Browser-certificate based login: MyOpenid. This is the most secure and most convenient method, as long and only as long as the authorized user is the only person with physical access to the computer with the browser, which contains that certificate.
  

Javascript filtering of user generated content and applications

Javascript is dangerous and common practice today, when the user is allowed to provide content, e.g. at the comment section of a blog, is to filter out at server side any possible Javascript elements in the HTML text the user provided in his comment. If interactivity is explicitly desired for user generated applications which run on the platform of the hosting application provider, then a subset of Javascript must be allowed, otherwise Web 2.0 user experience is gone. Facebook deals with this by defining its own Javascript subset called FBJS. Recently various technical approaches have been announced / discussed / released which deal with filtering of Javascript:

• AdSafe: A Javscript subset defined by Douglas Crockford. He is initially targeting the advertising industry with it, so that interactive ads can be placed on web pages without compromising the user's security. ADsafe can also be used for of mashup components such as widgets.
  
• Caja: From google. Does Javascript source-to-source translation. Currently written in Javascript, but the source code repository contains an empty folder titled "Java", so I guess that is what will come next.
  
• JStify: Filtering and also automatic replacement of unsecure Javascript code. Announced, no code released yet, written in Ocaml.
• and last and least my own not yet formally announced approach based on lexical analysis and parsing of Javascript in Erlang, as part of a Javascipt-to-Erlang compiler (to be open sourced) which I'm gonna use at the server side of my startup skast.com.

Mount Amazon S3 on your Mac

I recently discovered s3fs, a fuse based file system which allows to mount an Amazon S3 bucket like a normal file system at your PC. With other words, it is very easy and relatively cheap now to expand your local hard drive with terabytes of backup or Photo/Video storage. And with the recently announced Amazon S3 SLA, it seems Amazon is committed to continue with this service.
s3fs is currently only provided in form of source code. But with a little bit of tweaking I got it compiling and running on my Mac, see below the instructions and the required Mac specific modifications:

Get and install the latest MacFUSE-Core. This is just background process, without any
GUI elements.

Checkout the s3fs source code. Because it is just one file, you can even copy it manually in a newly created directory and add the following line to s3fs.cpp
(after #define FUSE_USE_VERSION 26):

 #define __off_t off_t

Start a terminal, change to the directory with the s3fs.cpp file inside and prepare the environment:

  export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig

Next modify the Makefile (or create one, if you just downloaded the s3fs.cpp):

  all:
  g++ -Wall -D__FreeBSD__=10 -D_FILE_OFFSET_BITS=64 $(shell pkg-config fuse --cflags --libs) -lcurl -lcrypto $(shell xml2-config --cflags --libs) -ggdb s3fs.cpp -o s3fs
@echo ok!

  clean:
  rm -f s3fs s3fs.o

If everything went well, you should have now a binary file s3fs in that directory. Now create a file /etc/passwd-s3fs which contains just your Amazon ID and secret key separated by ":", e.g:

  example-id:example-secret-key

now you create a new Amazon S3 bucket. I have been using the S3 Browser for that. Define a mount point, I just created a new directory in my home folder for that. Now you can mount that directory to your newly created Amazon S3 Bucket by running the following command:

  ./s3fs your-bucket-name your-mount-point-directory

Now you should see the MacFUSE icon in the Mac OS X Finder. And any file you put into your mount point directory is now physically stored at your bucket at Amazon S3. It's not very user friendly yet, let's hope the MacFusion guys integrate it soon into their excellent Fuse tool. I plan to integrate this S3 bucket access via s3fs into my software development toolchain. Because everything is scripted there, s3fs works fine for me, even if lots of important features are still missing.

Update:

In case you don't have libxml2 installed already, you need to install it first:

  sudo port install libxml2

Erlang refactoring with Emacs / Wrangler

With Wrangler, an extension to distel, one can now easily refactor Erlang programs with Emacs. Currently the following refactorings are supported:

• rename variable/function
• rename module
• generalize definition
• move a function definition to another module

At first attempt it seemed not to work, because it could not find header files from my application include directory, but after an e-mail exchange with the author I got it working, I just had to add the include directory path to list of source file directories known to wrangler.

Erlang compilation with Emacs

Since I switched to a Mac as my primary development platform I am using aquamacs for all my my coding needs. It is a fast native app, looks good, is highly customizable and together with distel it integrates very well with Erlang. Just one thing was giving me a headache:

Compiling from within Emacs based on an Emakefile

I tried some script by Alexey Lebedeff recently discussed on the Erlang mailing list, but at first try I did not manage to adapt the script to fit my needs. The situation got worse when I started to use non-Erlang source files such as templates and lexer / parser grammars which all get compiled to beam files and need to be reloaded when compilation was successful. After digging a bit into Emacs Lisp I came up with the following approach:

By pressing a function key (F13 in my case) emacs invokes via RPC at the Erlang application a custom compile command which in case of an Erlang source file switches to the directory of the Emakefile, runs a make:all([load]). and switches back to the original directory of the currently edited file. If the source file is a non-Erlang, than custom code gets called to perform all the necessary steps until beam file reloading.

There are two implications which need to be considered before adapting this approach:

• An application must contain specific code which can be called by emacs and that code must be able to determine the location of the Emakefile.
• The approach can't be used for initial compilation, because emacs calls a function from an application module, which must be compiled already.

And of course you must have distel installed. Below a code snippet showing the Emacs Lisp code, which goes into emacs.el (or Preferences.el in case of aquamacs):

(defun my-erlang-compile ()
(interactive)
(save-some-buffers (not compilation-ask-about-save) nil)
(save-excursion
(let ((thisdir default-directory))
(setq src-file-name buffer-file-name)
(with-current-buffer (get-buffer-create "*erl-output*")
(setq default-directory thisdir)
(erase-buffer)
(compilation-mode)
(toggle-read-only nil)
(setq compilation-current-error nil)
(display-buffer (current-buffer))
(erl-spawn
(erl-send-rpc (erl-target-node)
            'distel
            'eval_expression
            (list (format "myapp_make:all(%S)." src-file-name)))
(erl-receive ()
  ((['rex ['ok string]]
    (insert string))
   (['rex ['error reason]]
    (insert reason))
   (other
    (message "Unexpected: %S" other)))))))))

(add-hook 'erlang-mode-hook 'my-erlang-mode-hook)
(defun my-erlang-mode-hook ()
;; when starting an Erlang shell in Emacs, default in the node name
  (setq inferior-erlang-machine-options '("-sname" "emacs"))

(define-key erlang-mode-map [f13]
(lambda () (interactive)
(progn
(my-erlang-compile))))
)

;; just to illustrate how to use custom erlang compilation
;; from within other modes
(add-hook 'foo-helper-mode-hook
(lambda ()            
 (define-key foo-helper-mode-map [f13]
   (lambda () (interactive)
     (progn
       (my-erlang-compile))))))

With the elisp snippet from above, the corresponding Erlang application must contain a module myapp_make which implements the function all/1 :

all(Path) ->
case filename:extension(Path) of
".erl" -> 
 {ok, OldDir} = file:get_cwd(),
 ok = file:set_cwd(my_find_emakefile_dir()),
 make:all([load]),
 ok = file:set_cwd(OldDir);
Ext ->
 maybe_my_custom_stuff
end.

Funny Firefox icon

I really like this one ! I don't know who created the modification, so I can't give credit.

Mozilla dropping gecko for webkit ?

Probably not, because open source developers are traditionally often a bit opinated ...
But check out this post by AllPeers CTO Matthew Gertner, he thinks there are plenty of reasons to drop gecko for webkit, the biggest one Mozilla's large memory footprint. And of course there are also many drawbacks, most important one is loosing XUL and XPCOM. All the extensions which helped Firefox to gain momentum, would not be usable anymore at least until a webkit compatible XUL / XPCOM (if that is possible at all) is in place. In this context it is worth to mention that the post comes from an extension developer, so the drawbacks of his proposal would hurt himself as well !

VoIP audio codecs in next flashplayer

at least that is what somebody is blogging from the MAX 2007 Adobe conference:

====
Danielle Deibler to discuss VOIP in Flash Player.
They are a VOIP service - point to point media session, and peer to peer technology for Flash Player.
Also going to be enabling some extended codecs in the Flash Player beyond the ones available today.
...
====

The current audio / video codecs in the flashplayer all suck, they are either outdated in terms of performance (ADPCM, h263), are proprietary and with inexistent open source encoders (NellyMoser, VP6) or are patent encumbered (mp3, h264). Hope they get it right next time.

The Future of Firefox and Javascript

Here comes the open source answer from Mozilla hacker John Resig to the recent announcements from Adobe (see my previous post). And there are some interesting parallels between the two, e.g. Astro 3D Effects / Hydra and the Firefox 3D canvas which seems to be an OpenGL wrapper and allows direct embedding of native c-code. Let's hope this get's all shipped with Firefox 3 in the not so distant future !

Astro & Hydra - about the next flashplayer

Flash is a proprietary technology, often associated with "Skip Intro" and annoying ads. And for web applications most developers prefer AJAX. But for video and graphics effects there is nothing which can beat flash. And Adobe keeps making the player better and faster. They have yesterday updated their current flashplayer beta with some h264 video playback improvements and today revealed some of the capabilities of their next flashplayer codenamed Astro:

Advanced Text Layout
As long as the text is outside the Browser DOM, I am not really interested in that kind of stuff, sorry Adobe ...

3D Effects
There are today quite impressive 3D engines written in Actionscript such as Sandy and Papervision, but a native implementation will bring obviously a huge performance boost.

Custom Filters, Blend Modes and other Bitmap Effects
That seems to be something completely new. They provide a downloadable application called AIF Toolkit, (for Mac and Windows) where one can define graphical effects in a new scripting language called Hydra (syntax looks like a mixture between actionscript and c). I installed the tool on my Mac and loaded one of the provided code samples. The tool is a simple application with a text editor, an image preview area and a button labeled Run. If you press it, and the syntax of the code snippet is correct, the snippet gets compiled. The tool also contains a pdf document with the hydra language specification.
I couldn't figure out how to test such a compiled filter or effect. And I don't see how this can be integrated into an existing tool chain. But I guess this is not a problem for the average flash developer who is used to work with proprietary tools.

Update: Tried again and everything magically worked. After compilation you see immediately the result of the script in the image preview area and if the script has parameters, you can interactively change them. For example with the simple script below you can set the opacity of a bitmap:

// opacity:  A simple example to demonstrate the use of the hydra to change
// the opacity of an image based on an input parameter
kernel opacity
{
// An input parameter used to specify the opacity of the image.
    // The parameter comes directly from the user by way of the UI that gets
    // created for the filter.
    parameter float alpha;

// evaluatePixel(): The function of the filter that actually does the
    //                  processing of the image.  This function is called once
    //                  for each pixel of the output image.
    void
evaluatePixel(in image4 myImage, out float4 dst)
{
   // Acquire the pixel value from the image at the current location
        float4 myPixel = sampleNearest(myImage, outCoord());

   // Calculate the output the pixel value:
        dst = myPixel * alpha;
}
}

Douglas Crockford's proposal for solving mashup security problems

It's too late now to fix the problems of the internet with the browser. So Douglas (who works for yahoo) suggests to use google gears to solve cross domain security problems:

Internet platforms

I usually do not repeat what others blog about, but this post I believe is worth an exception. Marc Andreessen (Netscape founder and currently Ning CEO) coins the term of Internet platform of Level 1, 2 or 3. Marc says:

If you can program it, then it's a platform. If you can't, then it's not.

Level 1 represents REST and SOAP API's , where the core system is outside the API. Marc calls this the "Access API". The barriers to entry for app developers are high, because it requires a lot of technical expertise, IT infrastructure and financial resources to create, maintain and scale an app on the Level 2 platform.

Level 2 Marc describes as "Plug-In API". Facebook represents a Level 2 API. The developer creates an app which users can plug in into their Facebook profile. As with Level 1, the core system of the app lives outside of the platform, e.g. Facebook apps are hosted on the developer's server and not on Facebook's server farm. While applications running on the Level 2 platform are easier to build than applications for the Level 1 platform, it is much more difficult to build the platform itself.

Now comes the most interesting part, Level 3, what Marc calls "Runtime Environment". Everything lives inside the platform. Developers don't need to run their own servers anymore, they just upload their code to the platform. And again, it is getting more difficult to build the platform, but easier to build the apps running on that platform. Examples for Level 3 platforms are Ning, SalesForce and SecondLife.

And Marc says:

I believe that in the long run, all credible large-scale Internet companies will provide Level 3 platforms.

and it goes on and on, huge post, but really worth reading it and now I know how the thing I am secretly working on is called ...

Mochiweb - an erlang based webserver toolkit

Yesterday I asked on the erlang mailing list if somebody knows about a fast and simple HTTP server in Erlang, specifically suited to dynamic requests. And I got an answer from Bob Ippolito, providing even more than I was looking for: an easy extendable and highly configurable webserver toolkit, which contains everthing from static file serving to URL and multipart decoding to JSON handling. Below a sample Bob provided to show how to build a webserver using mochiweb:

-module(c4_http).
-export([start/0, loop/2, stop/0]).
-define(DEFAULTS, [{name, ?MODULE},
              {port, 9952}]).

start() ->
DocRoot = filename:dirname(filename:dirname(code:which(?MODULE))),
code:add_patha(filename:join([DocRoot, "mochiweb-c4", "ebin"])),
Loop = fun (Req) -> ?MODULE:loop(Req, DocRoot) end,
{ok, Pid} = c4_adder_otp:start(),
register(c4_hit_counter, Pid),
mochiweb_http:start([{loop, Loop} | ?DEFAULTS]).

stop() ->
c4_adder_otp:stop(c4_hit_counter),
mochiweb_http:stop(?MODULE).

loop(Req, DocRoot) ->
"/" ++ Path = Req:get(path),
Hits = c4_adder_otp:add(1, c4_hit_counter),
case Req:get(method) of
   M when M =:= 'GET'; M =:= 'HEAD' ->
       case Path of
           "timer" ->
               Response = Req:ok({"text/plain", chunked}),
               timer(Response);
           "static" ->
               Req:ok({"text/plain", "static response"});
           "hits" ->
               Req:ok({"text/plain",
                       io_lib:format("Hits: ~p~n", [Hits])});
           "nodes" ->
               Req:ok({"text/plain",
                       io_lib:format("~p~n", [nodes()])});
           "dump" ->
               Req:ok({"text/plain",
                       io_lib:format("~p~n", [Req:dump()])});
           _ ->
               Req:serve_file(Path, DocRoot)
       end;
   _ ->
       Req:respond({501, [], ""})
end.

timer(Req) ->
Req:write_chunk(io_lib:format("The time is: ~p~n",
                             [calendar:local_time()])),
timer:sleep(1000),
timer(Req).

Bob mentioned in the post, that he will eventually "propperly open source it", in the meantime, sources, examples and slides (Erlang introduction) can be found here.

ErlyVideo - RTMP / Flash streaming server

I resurrected my attempt of implementing a RTMP / Flash streaming server and turned it into an open source project: ErlyVideo. When I originally wrote that code, sometime last year, it mainly served as practical example for learning Erlang. Streaming actually worked, I could record and playback audio and video from the flashplayer, but the code was ugly, the TCP server was blocking and worst of all, the RTMP protocol is a proprietary thing from Adobe, so I did a clean room implementation, step by step, trial and error, without actually knowing the protocol when I started, so it had to end up in a mess.
Now I cleaned it up a bit and turned it into a non-blocking OTP server application, but it is still just a proof of concept ...

Another Erlang imaging library binding

A query at Google code reveals that there are now 72 Erlang tagged projects hosted there. One caught my attention: Erlmagic, an erlang mapping to the image magick library. It is probably not as fast as my own (not because of me, because of the underlying c lib), much lower level solution erlycairo, but seems to have a rich feature set, see demo image above.

Erlang Facebook API

While looking for online resources about Erlang / Facebook, I found out that Brian Fink has released his Erlang Implementation of the Facebook API as open source project at googlecode. Great.

Erlang JS compiler - dealing with objects

I did some more work on the Erlang based Javascript compiler proof-of-concept. Also set up a googlecode project named 'jserl', but not yet uploaded any code.
Javascript has some functional aspects, but is also object oriented. I have used message passing for modeling objects. For every object definition in Javascript a message loop gets injected into the generated Erlang code. And I also started to implement some native functions as provided by the Javascript Interpreter. So let's take a look at an example to see what currently works and how the generated Erlang code looks like:

Javascript code:

var argument_test = function (a) {
  var b = a + 1;
  return b;
};

var object_test = function (arg) {
  var obj = { property: 'text' };
  var test2 = obj.property;
  return test2;
};

var native_functions_test = function (arg) {
  return arg.length;
};

Compiled to a beam file this gives the following (expected and correct) results:

Eshell V5.5.5 (abort with ^G)
1> footest:argument_test(5).
6
2> footest:object_test(no_arg_dummy).
"text"
3> footest:native_functions_test("text").
4
4>

And here is how the Erlang source code (generated with erl_prettypr:format/1 from the abstract syntax tree) looks like:

-module(footest).
-compile(export_all).
argument_test(A) -> B = A + 1, B.
object_test(Arg) ->
Obj = obj_new("text"),
Test2 = case Obj of
 Int when is_integer(Int) -> "not defiend (yet ?)";
 Float when is_float(Float) -> "not defiend (yet ?)";
 List when is_list(List) ->
     case jserl:is_string(List) of
       true -> "not defiend (yet ?)";
       '_' -> "not defiend (yet ?)"
     end;
 Pid when is_pid(Pid) ->
     jserl:rpc(Pid, {get, 'Property'});
 '_' -> "not defiend (yet ?)"
end,
Test2.
native_functions_test(Arg) ->
case Arg of
Int when is_integer(Int) -> "not defiend (yet ?)";
Float when is_float(Float) -> "not defiend (yet ?)";
List when is_list(List) ->
case jserl:is_string(List) of
true -> length(Arg);
'_' -> "not defiend (yet ?)"
end;
Pid when is_pid(Pid) -> jserl:rpc(Pid, {get, 'Length'});
'_' -> "not defiend (yet ?)"
end.
obj(Property) ->
receive
{From, {get, 'Property'}} ->
From ! {self(), Property}, obj(Property);
{From, {set, 'Property', Val}} ->
From ! {self(), ok}, obj(Property);
{From, _Other} ->
From ! {self(), {err, no_such_member}}, obj(Property)
end.
obj_new(Property) -> spawn(fun () -> obj(Property) end).

Of course there is a lot of stuff missing, so it is currently more a pseudo script than an ECMA compliant Javascript compiler.

Erlang based Javascript compiler

There was recently a long thread on the Erlang mailing list about compiling Javascript to Erlang. Out of curiosity, how such an Erlang solution would compare to Rhino, which is a Javascript compiler for Java, I started to investigate in that field. First I tried my luck the traditional way with Lexer and Parser in Erlang. Thanks to Denis Loutrein, who had already written a LR Grammar file for yecc / leex, supporting a subset of Javascript, and who was so kind to share it with me, I had not to start at zero. Soon I realized that I did not "like" the complexity involved by that approach. Fortunately there exists another way, better suited to my needs, pioneered by Douglas Crockford: writing the parser in Javascript, using the simple but extremly efficient "Top Down Operator Precedence" method and then passing the parser as JSON object to Erlang, where the Javascript abstract syntax tree (AST) needs to be translated to an Erlang specific AST, which then can be compiled to Bytecode and loaded into the VM (or stored to a .beam file). I got very excited when I had a prototype working, which was running this simple JS function on the Erlang VM:

var foo = function (a) {
 var b = a + 1;
 return b;
};

First step was to adapt Crockford's parser to dojo, my preferred Javascript framework. Feeding the parser with the sample Javascript snippet form above results in an object, which can be JSON serialized to a pure AST representation:

{{"value",<<"=">>},
{"arity",<<"binary">>},
{"first",{{"value",<<"foo">>},{"arity",<<"name">>}}},
{"second",
{{"value",<<"function">>},
{"arity",<<"function">>},
{"first",[{{"value",<<"a">>},{"arity",<<"name">>}}]},
{"second",
[{{"value",<<"=">>},
{"arity",<<"binary">>},
{"first",
{{"value",<<"b">>},{"arity",<<"name">>}}},
{"second",
{{"value",<<"+">>},
{"arity",<<"binary">>},
{"first",
{{"value",<<"a">>},{"arity",<<"name">>}}},
{"second",
{{"value",1},{"arity",<<"literal">>}}}}}},
{{"value",<<"return">>},
{"arity",<<"statement">>},
{"first",
 {{"value",<<"b">>},{"arity",<<"name">>}}}}]}}}}

On the Erlang side, now comes the tricky part. The JSON tuple this needs to be translated into a tuple which represents an Erlang AST. For the example above my translator produces the following:

 {function,1,
  foo,
  1,
  [{clause,1,
           [{var,1,'A'}],
           [],
           [{match,1,
                   {var,1,'B'},
                   {op,1,'+',{var,1,'A'},{integer,1,1}}},
            {var,1,'B'}]}]}

Next, this tuple needs to be wrapped with some meta data attributes, such as module name and function exports:

[{attribute,1,module,footest},
{attribute,1,compile,export_all},
{function,1,
   foo,
   1,
   [{clause,1,
            [{var,1,'A'}],
            [],
            [{match,1,
                    {var,1,'B'},
                    {op,1,'+',{var,1,'A'},{integer,1,1}}},
             {var,1,'B'}]}]},
{eof,1}]

Now we have the complete module representation, and with compile:forms/1 it can be turned into byte code for directly loading into the VM (code:load_binary/3) or writing to a .beam file. It is even possible to get the Erlang representation of the source code with erl_prettypr:format/1:

"-module(footest)."
"-compile(export_all)."
"foo(A) -> B = A + 1, B."

What's next ?

Parser and translator currently only support a subset of the Javascript language. Here there is a lot to do. Another issue is the functional nature of Erlang. As long as I write functional Javascript, everything is fine. But somehow I also need to handle the non-functional aspects of Javascript. And setting up a project at googlecode for this.