Mochiweb - an erlang based webserver toolkit

Yesterday I asked on the erlang mailing list if somebody knows about a fast and simple HTTP server in Erlang, specifically suited to dynamic requests. And I got an answer from Bob Ippolito, providing even more than I was looking for: an easy extendable and highly configurable webserver toolkit, which contains everthing from static file serving to URL and multipart decoding to JSON handling. Below a sample Bob provided to show how to build a webserver using mochiweb:

-module(c4_http).
-export([start/0, loop/2, stop/0]).
-define(DEFAULTS, [{name, ?MODULE},
              {port, 9952}]).

start() ->
DocRoot = filename:dirname(filename:dirname(code:which(?MODULE))),
code:add_patha(filename:join([DocRoot, "mochiweb-c4", "ebin"])),
Loop = fun (Req) -> ?MODULE:loop(Req, DocRoot) end,
{ok, Pid} = c4_adder_otp:start(),
register(c4_hit_counter, Pid),
mochiweb_http:start([{loop, Loop} | ?DEFAULTS]).

stop() ->
c4_adder_otp:stop(c4_hit_counter),
mochiweb_http:stop(?MODULE).

loop(Req, DocRoot) ->
"/" ++ Path = Req:get(path),
Hits = c4_adder_otp:add(1, c4_hit_counter),
case Req:get(method) of
   M when M =:= 'GET'; M =:= 'HEAD' ->
       case Path of
           "timer" ->
               Response = Req:ok({"text/plain", chunked}),
               timer(Response);
           "static" ->
               Req:ok({"text/plain", "static response"});
           "hits" ->
               Req:ok({"text/plain",
                       io_lib:format("Hits: ~p~n", [Hits])});
           "nodes" ->
               Req:ok({"text/plain",
                       io_lib:format("~p~n", [nodes()])});
           "dump" ->
               Req:ok({"text/plain",
                       io_lib:format("~p~n", [Req:dump()])});
           _ ->
               Req:serve_file(Path, DocRoot)
       end;
   _ ->
       Req:respond({501, [], ""})
end.

timer(Req) ->
Req:write_chunk(io_lib:format("The time is: ~p~n",
                             [calendar:local_time()])),
timer:sleep(1000),
timer(Req).

Bob mentioned in the post, that he will eventually "propperly open source it", in the meantime, sources, examples and slides (Erlang introduction) can be found here.

ErlyVideo - RTMP / Flash streaming server

I resurrected my attempt of implementing a RTMP / Flash streaming server and turned it into an open source project: ErlyVideo. When I originally wrote that code, sometime last year, it mainly served as practical example for learning Erlang. Streaming actually worked, I could record and playback audio and video from the flashplayer, but the code was ugly, the TCP server was blocking and worst of all, the RTMP protocol is a proprietary thing from Adobe, so I did a clean room implementation, step by step, trial and error, without actually knowing the protocol when I started, so it had to end up in a mess.
Now I cleaned it up a bit and turned it into a non-blocking OTP server application, but it is still just a proof of concept ...

Another Erlang imaging library binding

A query at Google code reveals that there are now 72 Erlang tagged projects hosted there. One caught my attention: Erlmagic, an erlang mapping to the image magick library. It is probably not as fast as my own (not because of me, because of the underlying c lib), much lower level solution erlycairo, but seems to have a rich feature set, see demo image above.

Erlang Facebook API

While looking for online resources about Erlang / Facebook, I found out that Brian Fink has released his Erlang Implementation of the Facebook API as open source project at googlecode. Great.

Erlang JS compiler - dealing with objects

I did some more work on the Erlang based Javascript compiler proof-of-concept. Also set up a googlecode project named 'jserl', but not yet uploaded any code.
Javascript has some functional aspects, but is also object oriented. I have used message passing for modeling objects. For every object definition in Javascript a message loop gets injected into the generated Erlang code. And I also started to implement some native functions as provided by the Javascript Interpreter. So let's take a look at an example to see what currently works and how the generated Erlang code looks like:

Javascript code:

var argument_test = function (a) {
  var b = a + 1;
  return b;
};

var object_test = function (arg) {
  var obj = { property: 'text' };
  var test2 = obj.property;
  return test2;
};

var native_functions_test = function (arg) {
  return arg.length;
};

Compiled to a beam file this gives the following (expected and correct) results:

Eshell V5.5.5 (abort with ^G)
1> footest:argument_test(5).
6
2> footest:object_test(no_arg_dummy).
"text"
3> footest:native_functions_test("text").
4
4>

And here is how the Erlang source code (generated with erl_prettypr:format/1 from the abstract syntax tree) looks like:

-module(footest).
-compile(export_all).
argument_test(A) -> B = A + 1, B.
object_test(Arg) ->
Obj = obj_new("text"),
Test2 = case Obj of
 Int when is_integer(Int) -> "not defiend (yet ?)";
 Float when is_float(Float) -> "not defiend (yet ?)";
 List when is_list(List) ->
     case jserl:is_string(List) of
       true -> "not defiend (yet ?)";
       '_' -> "not defiend (yet ?)"
     end;
 Pid when is_pid(Pid) ->
     jserl:rpc(Pid, {get, 'Property'});
 '_' -> "not defiend (yet ?)"
end,
Test2.
native_functions_test(Arg) ->
case Arg of
Int when is_integer(Int) -> "not defiend (yet ?)";
Float when is_float(Float) -> "not defiend (yet ?)";
List when is_list(List) ->
case jserl:is_string(List) of
true -> length(Arg);
'_' -> "not defiend (yet ?)"
end;
Pid when is_pid(Pid) -> jserl:rpc(Pid, {get, 'Length'});
'_' -> "not defiend (yet ?)"
end.
obj(Property) ->
receive
{From, {get, 'Property'}} ->
From ! {self(), Property}, obj(Property);
{From, {set, 'Property', Val}} ->
From ! {self(), ok}, obj(Property);
{From, _Other} ->
From ! {self(), {err, no_such_member}}, obj(Property)
end.
obj_new(Property) -> spawn(fun () -> obj(Property) end).

Of course there is a lot of stuff missing, so it is currently more a pseudo script than an ECMA compliant Javascript compiler.

Erlang based Javascript compiler

There was recently a long thread on the Erlang mailing list about compiling Javascript to Erlang. Out of curiosity, how such an Erlang solution would compare to Rhino, which is a Javascript compiler for Java, I started to investigate in that field. First I tried my luck the traditional way with Lexer and Parser in Erlang. Thanks to Denis Loutrein, who had already written a LR Grammar file for yecc / leex, supporting a subset of Javascript, and who was so kind to share it with me, I had not to start at zero. Soon I realized that I did not "like" the complexity involved by that approach. Fortunately there exists another way, better suited to my needs, pioneered by Douglas Crockford: writing the parser in Javascript, using the simple but extremly efficient "Top Down Operator Precedence" method and then passing the parser as JSON object to Erlang, where the Javascript abstract syntax tree (AST) needs to be translated to an Erlang specific AST, which then can be compiled to Bytecode and loaded into the VM (or stored to a .beam file). I got very excited when I had a prototype working, which was running this simple JS function on the Erlang VM:

var foo = function (a) {
 var b = a + 1;
 return b;
};

First step was to adapt Crockford's parser to dojo, my preferred Javascript framework. Feeding the parser with the sample Javascript snippet form above results in an object, which can be JSON serialized to a pure AST representation:

{{"value",<<"=">>},
{"arity",<<"binary">>},
{"first",{{"value",<<"foo">>},{"arity",<<"name">>}}},
{"second",
{{"value",<<"function">>},
{"arity",<<"function">>},
{"first",[{{"value",<<"a">>},{"arity",<<"name">>}}]},
{"second",
[{{"value",<<"=">>},
{"arity",<<"binary">>},
{"first",
{{"value",<<"b">>},{"arity",<<"name">>}}},
{"second",
{{"value",<<"+">>},
{"arity",<<"binary">>},
{"first",
{{"value",<<"a">>},{"arity",<<"name">>}}},
{"second",
{{"value",1},{"arity",<<"literal">>}}}}}},
{{"value",<<"return">>},
{"arity",<<"statement">>},
{"first",
 {{"value",<<"b">>},{"arity",<<"name">>}}}}]}}}}

On the Erlang side, now comes the tricky part. The JSON tuple this needs to be translated into a tuple which represents an Erlang AST. For the example above my translator produces the following:

 {function,1,
  foo,
  1,
  [{clause,1,
           [{var,1,'A'}],
           [],
           [{match,1,
                   {var,1,'B'},
                   {op,1,'+',{var,1,'A'},{integer,1,1}}},
            {var,1,'B'}]}]}

Next, this tuple needs to be wrapped with some meta data attributes, such as module name and function exports:

[{attribute,1,module,footest},
{attribute,1,compile,export_all},
{function,1,
   foo,
   1,
   [{clause,1,
            [{var,1,'A'}],
            [],
            [{match,1,
                    {var,1,'B'},
                    {op,1,'+',{var,1,'A'},{integer,1,1}}},
             {var,1,'B'}]}]},
{eof,1}]

Now we have the complete module representation, and with compile:forms/1 it can be turned into byte code for directly loading into the VM (code:load_binary/3) or writing to a .beam file. It is even possible to get the Erlang representation of the source code with erl_prettypr:format/1:

"-module(footest)."
"-compile(export_all)."
"foo(A) -> B = A + 1, B."

What's next ?

Parser and translator currently only support a subset of the Javascript language. Here there is a lot to do. Another issue is the functional nature of Erlang. As long as I write functional Javascript, everything is fine. But somehow I also need to handle the non-functional aspects of Javascript. And setting up a project at googlecode for this.

Javascript based code editor

I was searching for a code editor written in JavaScript and capable of edit (and syntax highlight) source files in JavaScript, HTML and CSS. There are several active projects around dealing with this kind of stuff. After playing with the online demos and a quick look at the source and implementation details, here a summary of my three favorites:

CodePress
Rich feature set, support for many languages, does not work on Safari, LGPL licensed, not-so-active development: last commit about a month ago.

jsvi (vi-clone)
Impressive demo, seems to work on most browsers, but the code is copyrighted and without open source license.

9ne (emacs-clone)
Really impressive project by Rob Rohan, but after open sourcing (GPL) it about two and a half months ago, the source code repository shows no further commits. On Safari 3beta I couldn't do the most basic thing: inserting text. Otherwise and on Firefox it really feels like emacs.

Building cross-platform VideoFox (Firefox 3 alpha pre-8 with video patches)

What caused major headaches in the past (especially if you want to do it from the same source tree) has become pretty easy and straight forward these days, at least if you are using a Mac as primary development platform and run Windows and Linux in virtual machines. If you are just curious about the video enabled Firefox, go to Chris Double's testpage and download a binary release for your platform. If your platform is not supported yet, you want to learn the internals of firefox, or build a firefox extension with native components (that's my case, more about it later ...) then the following step-by-step instructions might help:

• Install are the required tools on your mac
  
• Install git on mac (package git-core from macports)
• clone the git repo form Chris:

git clone git://double.co.nz/git/firefox.git

• cd to the mozilla directory, create a .mozconfig and run the build command:

make -f client.mk build

• after some time, and if nothing goes wrong you should have now a firefox binary
• To build on windows, first share the folder with the cloned git repository in Parallels.
• Start Windows XP as Parallels VM and install the mozilla build tools
• Also install the free Microsoft Visual C++IDE and the Microsoft platform SDK
  
• Map the shared folder to a network drive (Context menu).
• Mount in C:\mozilla-build\msys\etc\fstab the network drive
• Reboot windows to activate the mount point
• Start the shell:

C:\mozilla-build\start-msvc8

• Go to your mozilla source directory and run the build command again.

make -f client.mk build

• Now it takes even longer to build, because of the Parallels VM overhead (vmware fusion might be faster ???)
• Next comes the Linux build, but I am still waiting at 14) !!

Update:
after about 8 hours the windows build was done (on mac it was just one hour). Unfortunately the Windows XP Firefox crashed when trying to run.

Update II:
The Windows XP build did not crash, it just took extremely long to start, I guess because of the Parallels network drive.

Update III:
Now the Linux build: there are three possibilities to access the source tree from Linux: NFS, Samba and FUSE-based sshfs. Because the first two options require a lot of tricky configuration, I went for the last, below the few steps required (on Ubuntu Feisty):

• Allow Remote Login on Mac (System Preferences > Sharing)
  
• Start Linux VM
  
• On Linux, install sshfs:

sudo apt-get install sshfs

• Add user to fuse group:

sudo adduser your-linux-username fuse

• Log off and on again to activate the group setting
• create a directory:

mkdir firefox-source

• mount the mac filesystem:
  

sudo sshfs your-mac-username@mac-IP:/Path/to/firefox-source /LinuxPath/to/firefox-source -o allow_other

• build it again (should be straight forward on Linux, haven't done that yet)
• To unmount:
  

sudo fusermount -u /LinuxPath/to/firefox-source

Messing around with code and colors

Inspired by this article I put together an erlang function to convert colors form HSL to RGB model. I will use it with the erlycairo imaging library and dynamic CSS generation as supported in erlyweb to get the colors right, hopefully !

Update:

How do erlang code snippets look like, just copy-pasted into google blogger and converted to courier font ? They get screwed up, because some special characters need proper escaping. Fortunately aquamacs, my preferred editor on the mac, has the htmlize mode, which turns anything into HTML. And also had to add some CSS from the generated emacs HTML output to the blogger template to make the snippet look more readable .

hsl_to_rgb(_H, S, L) when S == 0.0 ->
[L, L, L];

hsl_to_rgb(H, S, L) ->
Q = q(H,
S, L),
P = 2.0 * L - Q,
H1 = H / 360.0,
TList = [ t(T) || T <- [H1 + 1.0 / 3.0, H1, H1 - 1.0 / 3.0]],
[ c(P, Q, T) || T <- TList ].


q(_H, S, L) when (L < 0.5) ->
L * (1.0 + S);
q(_H, S, L) ->
L + S - L * S.

t(T) when (T < 0) ->
T + 1.0;
t(T) when (T > 1.0) ->
T - 1.0;
t(T) ->
T.

c(P, Q, T) when T < 1.0 / 6.0 ->
P + ((Q - P) * 6.0 * T);
c(P, Q, T) when 1.0 / 6.0 =< T, T < 0.5 ->
Q;
c(P, Q, T) when 0.5 =< T, T < 2.0 / 3.0 ->
P + ((Q - P) * (2.0 / 3.0 - T) * 6.0);
c(P, Q, T) ->
P.

Erlang blogs

Just while resurrecting this blog (take a look at the date of the first entry), I came across some other blogs of Erlang enthusiasts:

• Dive into Erlang
• Language Weenie
• archaelus
  

Erlycairo - an Erlang 2D graphics library

I just uploaded Erlang bindings for the cairo 2D graphics library to googlecode:

http://code.google.com/p/erlycairo/

I use it in my projects for generating CSS background images for flexible-size buttons, rounded corners and similar.

hello world