The Case for the Open Web

Alex Russel, dojo toolkit project leader, presented recently the slides below and I spotted "Web 3.0" on the cover slide ...

Prism - Turn any web page into a desktop app

Today Mozilla Labs launched Prism, a tool for easily turning web pages into desktop apps with their own window, desktop shortcut but no URL address bar (everything customizable, of course). It's not a new project, just the relaunch of what was previously mostly unknown as WebRunner. If the Mozilla guys do it right this time, Prism will compete head-to-head with Adobe AIR.

An early alpha version, currently only for windows, is available for download. I tried it out, for this blog, it is very easy to use, just one dialog for defining Name, URL and optional properties for the web page to be turned into a desktop app.

Google AdSense might mess up your page

Javascript expert Andrea Giammarchi points to some potential flaw in Google Adense. If you put on the same page Google AdSense and your own or other third party Javascripts, AdSense could mess with the others in a destructive way so that nothing works anymore on that page. The origin of the problem is that AdSense deletes properties of global objects, without checking whether these properties actually belong to AdSense.

What can one do to prevent problems ? The usual recommendation for dealing with the insecure aspects of Javascript: don't use global variables and put your stuff in your own namespace. See also my previous posts about Javascript security.

HTML5 movie trailer

Mixing HTTP and HTTPS without getting browser warnings

Great tip by Ned Batchelder: On HTTPS sites you can link to HTTP assets without getting a browser warning by using instead of

  http://fast.cdn.net/pix/smiley.jpg

the following syntax:

  //fast.cdn.net/pix/smiley.jpg

Why Erlang ?

Other functional languages such as Haskell or OCaml are for most people just a bit too academic. RubyOnRails is reaching end-of-hype, it's underlying language Ruby is dog slow and not scalable. Java is far too complicated for me and without blooooated tools such as eclipse or IDEA, it is a just mess to deal with, at least that was my painful experience I made recently: I developed a prototype of a Flash video streaming server in Erlang. There exists a more complete Java open source equivalent: Red5. I have a couple of years of experience as Java developer, so I thought the easiest was just to analyze the Java source code and port it to Erlang. Wrong. The Java code is so complex, that is was easier for me to reverse engineer the proprietary RTMP video streaming protocol by analyzing the TCP/IP packet flow of a running Red5 instance by using a network sniffer instead of reading its source code. Is there any other language I think one should not choose instead of Erlang ? Python. I actually had only heard good things about it, until I got to know about its dictated indentation style, which put an abrupt end for me to any further digging into that otherwise probably great language ...

Ok, end of ranting. Developers are a highly opinionated species. Sometimes discussions among developers about the right programming language take amusant, ridiculous and even religious dimensions, despite everybody trying to be objective (well, expect for his/her preferred language). So just use the right tool for the right job and don't listen to me... But listen to what the really smart and experienced people have to say. There is an interesting thread currently going on at the Erlang mailing list. A Java coder asks whether he should learn Erlang or OCaml next. Bob Ippolito (recently interviewed) says:

When I was evaluating Python alternatives for building the core
technology behind MochiAds I tried out a bunch of languages and Erlang
was the only one that was easy for me to learn and had the right
balance of features, performance, and reliability. A year later we
have about 16 machines running 80 Erlang nodes powering about 16
different "components" of our infrastructure and 4 people working on
it at the moment (originally it was just me). It worked out so well
that we rewrote the server component of our MochiBot service in Erlang
and we've been using it to build lots of internal tools such as our
monitoring software, our single sign-on service, etc. as well. None of
us had previous Erlang experience, but we're all very comfortable with
it now.

After about a year with Erlang, I'm not sure I could part with hot
code loading, light-weight processes, and multiplexed socket IO for
writing servers. Also, Mnesia has been really useful to us to
temporarily store "real-time" data (ram_copies) so that we don't have
to make users wait for it to get batched into the SQL databases. The
distribution stuff mostly Just Works once you figure out how to set it
up (though we did have one bad experience with a network partition due
to a switch acting up, it was recoverable manually).

O'Caml is a useful language too, but for writing a network app I can't
really imagine going with anything but Erlang if you're looking for
redundancy and scale. Unless you want to write your own half-baked
Erlang-like system before even trying to solve something a little
closer to your actual problem domain.

Joe Armstrong, one of the original authors of the Erlang language and Programming Erlang author says:

Erlang/OCaml/Haskell belong to the same language family - if you learn any one
of them then learning the next one in the family will be a lot easier
than starting from scratch.

These language differ - but have the same core concepts - the idea of immutable
state - programming with immutable state is the thing that you need to learn.
The details of how you do this vary from language to language (you can
use processes
with tail recursion to model state in Erlang, or monads in haskell, etc.).

I'd start with the language that most suits your problem domain - a
rough guess might
be to think of these languages as follows:

OCaml - use as a replacement for C - good for implementing virtual
machine emulators
tightly coded non-distributed applications.

Erlang - use as a replacement for Java - good for programming
distributed fault-tolerant
applications - good support for multicores/concurrency. Good as a glue
language to
glue together components co-ordinate activities on different machines etc.

Haskell - use for implementing domain specific languages, symbolic
computations etc.

And what has Erlang in the bag for Web developers ?

Not so much yet, if you look for an easy-to-learn, convention-over-configuration one-size-fits-all framework a la RubyOnRails. Web companies use Erlang today to overcome scalability problems for web based instant messaging (e.g.: ejabberd at twitter and Meebo). Among the few publicly known partially-to-mostly Erlang powered sites are MochiAds and Slideshare.

Here a little overview about some Erlang based web servers and frameworks:

• Yaws, the most popular Erlang web server. Active development since 2002. Many contributors, lots of add-ons. If you look for an Erlang-based, Apache-like web server, than yaws is the right thing for you.
  
• Erlyweb, by Yariv Sadan. The most popular Erlang MVC framework, built on top of yaws, enables you to easily do any-web-thing you can imagine, if you are a comfortable with Erlang and don' t mind to integrate yourself the AJAX toolkit of your choice. But don't expect a learning curve as with RubyOnRails, where you can start in the morning, without ever having heard anything about Ruby before, and at night you have your first simple web app running and have learnt Ruby without even noticing it.
  
• Tercio, by Eric Merrit. Different philosophy than Erlyweb, targeting AJAX apps which do most or even all rendering at client side.
  
• Mochiweb by Bob Ippolito. My preferred toolkit to easily build a custom HTTP server.
  
• And last and least, a shameless plug for my own upcoming web framework and service, which aims to lower the barrier to entry for individuals and companies doing utility computing based development and hosting of scalable AJAX / Comet web apps. You won't even need to know Erlang to start with, unless you want to customize the framework itself. More about this when I actually have something to show ...
  

Amazon EC2 instances now with up to 15 GB RAM

Utility computing is getting more and more interesting for scalable webhosting. So far Amazon EC2 only had one instance type with 1.7 GB RAM, 160 GB HD, 32-bit platform and $0.10 per instance hour. Now there are two new Amazon EC2 instance type, both 64-bit platform and with significantly more power:

Large instance (new):
7.5 GB RAM, 850 GB HD, four times more computing performance, $0.40 per instance hour

Extra large instance (new):
15 GB RAM, 1690 GB HD, eight times more computing performance, $0.80 per instance hour

I am working right now on some Erlang tools to simplify web application hosting on utility computing infrastructure and look forward to get my hands dirty on this new EC2 instances !

Improving web application security without degrading user experience

Generally speaking, web application security involves a trade-off between security and the user's convenience. But I am not generally speaking in this article. I am trying to explore some areas where it is possible to improve security and let equal or even improve the user experience.

Single-Sign-On with OpenID based authentication

First, this makes live easier for the application developer, because to start with he has only to implement a client library (available for PHP, Ruby, Python, ...) at his server back end and not a whole authentication server. The end user gets a Single-Sign-On and he can chose from any public OpenId provider he wants. That's the theory. In practice, OpenID has been something new, just confusing the user. But this is changing rapidly, according to this document, the adoption rate is exponential and there are already 120 Mio OpenIdDs out there.
Because OpenID decouples the choice for an authentication server from the web application itself, the user can minimize the trade-off between security and convenience purely based on his preferences or his particular situation. And now there exist also innovative, password-less approaches:

• Image-sequence based login: Vidoop. Instead of a password, the user needs to remember an image sequence. There are quite a few interesting facts about Vidoop: e.g. it can generate ad-revenue from the images for the Web application provider, it is resistant against repetitive logins with a stolen token (the image sequence) because it is based on a challenge-response method which is different for every login attempt.
  
• Browser-certificate based login: MyOpenid. This is the most secure and most convenient method, as long and only as long as the authorized user is the only person with physical access to the computer with the browser, which contains that certificate.
  

Javascript filtering of user generated content and applications

Javascript is dangerous and common practice today, when the user is allowed to provide content, e.g. at the comment section of a blog, is to filter out at server side any possible Javascript elements in the HTML text the user provided in his comment. If interactivity is explicitly desired for user generated applications which run on the platform of the hosting application provider, then a subset of Javascript must be allowed, otherwise Web 2.0 user experience is gone. Facebook deals with this by defining its own Javascript subset called FBJS. Recently various technical approaches have been announced / discussed / released which deal with filtering of Javascript:

• AdSafe: A Javscript subset defined by Douglas Crockford. He is initially targeting the advertising industry with it, so that interactive ads can be placed on web pages without compromising the user's security. ADsafe can also be used for of mashup components such as widgets.
  
• Caja: From google. Does Javascript source-to-source translation. Currently written in Javascript, but the source code repository contains an empty folder titled "Java", so I guess that is what will come next.
  
• JStify: Filtering and also automatic replacement of unsecure Javascript code. Announced, no code released yet, written in Ocaml.
• and last and least my own not yet formally announced approach based on lexical analysis and parsing of Javascript in Erlang, as part of a Javascipt-to-Erlang compiler (to be open sourced) which I'm gonna use at the server side of my startup skast.com.

Mount Amazon S3 on your Mac

I recently discovered s3fs, a fuse based file system which allows to mount an Amazon S3 bucket like a normal file system at your PC. With other words, it is very easy and relatively cheap now to expand your local hard drive with terabytes of backup or Photo/Video storage. And with the recently announced Amazon S3 SLA, it seems Amazon is committed to continue with this service.
s3fs is currently only provided in form of source code. But with a little bit of tweaking I got it compiling and running on my Mac, see below the instructions and the required Mac specific modifications:

Get and install the latest MacFUSE-Core. This is just background process, without any
GUI elements.

Checkout the s3fs source code. Because it is just one file, you can even copy it manually in a newly created directory and add the following line to s3fs.cpp
(after #define FUSE_USE_VERSION 26):

 #define __off_t off_t

Start a terminal, change to the directory with the s3fs.cpp file inside and prepare the environment:

  export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig

Next modify the Makefile (or create one, if you just downloaded the s3fs.cpp):

  all:
  g++ -Wall -D__FreeBSD__=10 -D_FILE_OFFSET_BITS=64 $(shell pkg-config fuse --cflags --libs) -lcurl -lcrypto $(shell xml2-config --cflags --libs) -ggdb s3fs.cpp -o s3fs
@echo ok!

  clean:
  rm -f s3fs s3fs.o

If everything went well, you should have now a binary file s3fs in that directory. Now create a file /etc/passwd-s3fs which contains just your Amazon ID and secret key separated by ":", e.g:

  example-id:example-secret-key

now you create a new Amazon S3 bucket. I have been using the S3 Browser for that. Define a mount point, I just created a new directory in my home folder for that. Now you can mount that directory to your newly created Amazon S3 Bucket by running the following command:

  ./s3fs your-bucket-name your-mount-point-directory

Now you should see the MacFUSE icon in the Mac OS X Finder. And any file you put into your mount point directory is now physically stored at your bucket at Amazon S3. It's not very user friendly yet, let's hope the MacFusion guys integrate it soon into their excellent Fuse tool. I plan to integrate this S3 bucket access via s3fs into my software development toolchain. Because everything is scripted there, s3fs works fine for me, even if lots of important features are still missing.

Update:

In case you don't have libxml2 installed already, you need to install it first:

  sudo port install libxml2

Erlang refactoring with Emacs / Wrangler

With Wrangler, an extension to distel, one can now easily refactor Erlang programs with Emacs. Currently the following refactorings are supported:

• rename variable/function
• rename module
• generalize definition
• move a function definition to another module

At first attempt it seemed not to work, because it could not find header files from my application include directory, but after an e-mail exchange with the author I got it working, I just had to add the include directory path to list of source file directories known to wrangler.

Erlang compilation with Emacs

Since I switched to a Mac as my primary development platform I am using aquamacs for all my my coding needs. It is a fast native app, looks good, is highly customizable and together with distel it integrates very well with Erlang. Just one thing was giving me a headache:

Compiling from within Emacs based on an Emakefile

I tried some script by Alexey Lebedeff recently discussed on the Erlang mailing list, but at first try I did not manage to adapt the script to fit my needs. The situation got worse when I started to use non-Erlang source files such as templates and lexer / parser grammars which all get compiled to beam files and need to be reloaded when compilation was successful. After digging a bit into Emacs Lisp I came up with the following approach:

By pressing a function key (F13 in my case) emacs invokes via RPC at the Erlang application a custom compile command which in case of an Erlang source file switches to the directory of the Emakefile, runs a make:all([load]). and switches back to the original directory of the currently edited file. If the source file is a non-Erlang, than custom code gets called to perform all the necessary steps until beam file reloading.

There are two implications which need to be considered before adapting this approach:

• An application must contain specific code which can be called by emacs and that code must be able to determine the location of the Emakefile.
• The approach can't be used for initial compilation, because emacs calls a function from an application module, which must be compiled already.

And of course you must have distel installed. Below a code snippet showing the Emacs Lisp code, which goes into emacs.el (or Preferences.el in case of aquamacs):

(defun my-erlang-compile ()
(interactive)
(save-some-buffers (not compilation-ask-about-save) nil)
(save-excursion
(let ((thisdir default-directory))
(setq src-file-name buffer-file-name)
(with-current-buffer (get-buffer-create "*erl-output*")
(setq default-directory thisdir)
(erase-buffer)
(compilation-mode)
(toggle-read-only nil)
(setq compilation-current-error nil)
(display-buffer (current-buffer))
(erl-spawn
(erl-send-rpc (erl-target-node)
            'distel
            'eval_expression
            (list (format "myapp_make:all(%S)." src-file-name)))
(erl-receive ()
  ((['rex ['ok string]]
    (insert string))
   (['rex ['error reason]]
    (insert reason))
   (other
    (message "Unexpected: %S" other)))))))))

(add-hook 'erlang-mode-hook 'my-erlang-mode-hook)
(defun my-erlang-mode-hook ()
;; when starting an Erlang shell in Emacs, default in the node name
  (setq inferior-erlang-machine-options '("-sname" "emacs"))

(define-key erlang-mode-map [f13]
(lambda () (interactive)
(progn
(my-erlang-compile))))
)

;; just to illustrate how to use custom erlang compilation
;; from within other modes
(add-hook 'foo-helper-mode-hook
(lambda ()            
 (define-key foo-helper-mode-map [f13]
   (lambda () (interactive)
     (progn
       (my-erlang-compile))))))

With the elisp snippet from above, the corresponding Erlang application must contain a module myapp_make which implements the function all/1 :

all(Path) ->
case filename:extension(Path) of
".erl" -> 
 {ok, OldDir} = file:get_cwd(),
 ok = file:set_cwd(my_find_emakefile_dir()),
 make:all([load]),
 ok = file:set_cwd(OldDir);
Ext ->
 maybe_my_custom_stuff
end.

Funny Firefox icon

I really like this one ! I don't know who created the modification, so I can't give credit.

Mozilla dropping gecko for webkit ?

Probably not, because open source developers are traditionally often a bit opinated ...
But check out this post by AllPeers CTO Matthew Gertner, he thinks there are plenty of reasons to drop gecko for webkit, the biggest one Mozilla's large memory footprint. And of course there are also many drawbacks, most important one is loosing XUL and XPCOM. All the extensions which helped Firefox to gain momentum, would not be usable anymore at least until a webkit compatible XUL / XPCOM (if that is possible at all) is in place. In this context it is worth to mention that the post comes from an extension developer, so the drawbacks of his proposal would hurt himself as well !

VoIP audio codecs in next flashplayer

at least that is what somebody is blogging from the MAX 2007 Adobe conference:

====
Danielle Deibler to discuss VOIP in Flash Player.
They are a VOIP service - point to point media session, and peer to peer technology for Flash Player.
Also going to be enabling some extended codecs in the Flash Player beyond the ones available today.
...
====

The current audio / video codecs in the flashplayer all suck, they are either outdated in terms of performance (ADPCM, h263), are proprietary and with inexistent open source encoders (NellyMoser, VP6) or are patent encumbered (mp3, h264). Hope they get it right next time.

The Future of Firefox and Javascript

Here comes the open source answer from Mozilla hacker John Resig to the recent announcements from Adobe (see my previous post). And there are some interesting parallels between the two, e.g. Astro 3D Effects / Hydra and the Firefox 3D canvas which seems to be an OpenGL wrapper and allows direct embedding of native c-code. Let's hope this get's all shipped with Firefox 3 in the not so distant future !

Astro & Hydra - about the next flashplayer

Flash is a proprietary technology, often associated with "Skip Intro" and annoying ads. And for web applications most developers prefer AJAX. But for video and graphics effects there is nothing which can beat flash. And Adobe keeps making the player better and faster. They have yesterday updated their current flashplayer beta with some h264 video playback improvements and today revealed some of the capabilities of their next flashplayer codenamed Astro:

Advanced Text Layout
As long as the text is outside the Browser DOM, I am not really interested in that kind of stuff, sorry Adobe ...

3D Effects
There are today quite impressive 3D engines written in Actionscript such as Sandy and Papervision, but a native implementation will bring obviously a huge performance boost.

Custom Filters, Blend Modes and other Bitmap Effects
That seems to be something completely new. They provide a downloadable application called AIF Toolkit, (for Mac and Windows) where one can define graphical effects in a new scripting language called Hydra (syntax looks like a mixture between actionscript and c). I installed the tool on my Mac and loaded one of the provided code samples. The tool is a simple application with a text editor, an image preview area and a button labeled Run. If you press it, and the syntax of the code snippet is correct, the snippet gets compiled. The tool also contains a pdf document with the hydra language specification.
I couldn't figure out how to test such a compiled filter or effect. And I don't see how this can be integrated into an existing tool chain. But I guess this is not a problem for the average flash developer who is used to work with proprietary tools.

Update: Tried again and everything magically worked. After compilation you see immediately the result of the script in the image preview area and if the script has parameters, you can interactively change them. For example with the simple script below you can set the opacity of a bitmap:

// opacity:  A simple example to demonstrate the use of the hydra to change
// the opacity of an image based on an input parameter
kernel opacity
{
// An input parameter used to specify the opacity of the image.
    // The parameter comes directly from the user by way of the UI that gets
    // created for the filter.
    parameter float alpha;

// evaluatePixel(): The function of the filter that actually does the
    //                  processing of the image.  This function is called once
    //                  for each pixel of the output image.
    void
evaluatePixel(in image4 myImage, out float4 dst)
{
   // Acquire the pixel value from the image at the current location
        float4 myPixel = sampleNearest(myImage, outCoord());

   // Calculate the output the pixel value:
        dst = myPixel * alpha;
}
}

Douglas Crockford's proposal for solving mashup security problems

It's too late now to fix the problems of the internet with the browser. So Douglas (who works for yahoo) suggests to use google gears to solve cross domain security problems:

Internet platforms

I usually do not repeat what others blog about, but this post I believe is worth an exception. Marc Andreessen (Netscape founder and currently Ning CEO) coins the term of Internet platform of Level 1, 2 or 3. Marc says:

If you can program it, then it's a platform. If you can't, then it's not.

Level 1 represents REST and SOAP API's , where the core system is outside the API. Marc calls this the "Access API". The barriers to entry for app developers are high, because it requires a lot of technical expertise, IT infrastructure and financial resources to create, maintain and scale an app on the Level 2 platform.

Level 2 Marc describes as "Plug-In API". Facebook represents a Level 2 API. The developer creates an app which users can plug in into their Facebook profile. As with Level 1, the core system of the app lives outside of the platform, e.g. Facebook apps are hosted on the developer's server and not on Facebook's server farm. While applications running on the Level 2 platform are easier to build than applications for the Level 1 platform, it is much more difficult to build the platform itself.

Now comes the most interesting part, Level 3, what Marc calls "Runtime Environment". Everything lives inside the platform. Developers don't need to run their own servers anymore, they just upload their code to the platform. And again, it is getting more difficult to build the platform, but easier to build the apps running on that platform. Examples for Level 3 platforms are Ning, SalesForce and SecondLife.

And Marc says:

I believe that in the long run, all credible large-scale Internet companies will provide Level 3 platforms.

and it goes on and on, huge post, but really worth reading it and now I know how the thing I am secretly working on is called ...

Mochiweb - an erlang based webserver toolkit

Yesterday I asked on the erlang mailing list if somebody knows about a fast and simple HTTP server in Erlang, specifically suited to dynamic requests. And I got an answer from Bob Ippolito, providing even more than I was looking for: an easy extendable and highly configurable webserver toolkit, which contains everthing from static file serving to URL and multipart decoding to JSON handling. Below a sample Bob provided to show how to build a webserver using mochiweb:

-module(c4_http).
-export([start/0, loop/2, stop/0]).
-define(DEFAULTS, [{name, ?MODULE},
              {port, 9952}]).

start() ->
DocRoot = filename:dirname(filename:dirname(code:which(?MODULE))),
code:add_patha(filename:join([DocRoot, "mochiweb-c4", "ebin"])),
Loop = fun (Req) -> ?MODULE:loop(Req, DocRoot) end,
{ok, Pid} = c4_adder_otp:start(),
register(c4_hit_counter, Pid),
mochiweb_http:start([{loop, Loop} | ?DEFAULTS]).

stop() ->
c4_adder_otp:stop(c4_hit_counter),
mochiweb_http:stop(?MODULE).

loop(Req, DocRoot) ->
"/" ++ Path = Req:get(path),
Hits = c4_adder_otp:add(1, c4_hit_counter),
case Req:get(method) of
   M when M =:= 'GET'; M =:= 'HEAD' ->
       case Path of
           "timer" ->
               Response = Req:ok({"text/plain", chunked}),
               timer(Response);
           "static" ->
               Req:ok({"text/plain", "static response"});
           "hits" ->
               Req:ok({"text/plain",
                       io_lib:format("Hits: ~p~n", [Hits])});
           "nodes" ->
               Req:ok({"text/plain",
                       io_lib:format("~p~n", [nodes()])});
           "dump" ->
               Req:ok({"text/plain",
                       io_lib:format("~p~n", [Req:dump()])});
           _ ->
               Req:serve_file(Path, DocRoot)
       end;
   _ ->
       Req:respond({501, [], ""})
end.

timer(Req) ->
Req:write_chunk(io_lib:format("The time is: ~p~n",
                             [calendar:local_time()])),
timer:sleep(1000),
timer(Req).

Bob mentioned in the post, that he will eventually "propperly open source it", in the meantime, sources, examples and slides (Erlang introduction) can be found here.

ErlyVideo - RTMP / Flash streaming server

I resurrected my attempt of implementing a RTMP / Flash streaming server and turned it into an open source project: ErlyVideo. When I originally wrote that code, sometime last year, it mainly served as practical example for learning Erlang. Streaming actually worked, I could record and playback audio and video from the flashplayer, but the code was ugly, the TCP server was blocking and worst of all, the RTMP protocol is a proprietary thing from Adobe, so I did a clean room implementation, step by step, trial and error, without actually knowing the protocol when I started, so it had to end up in a mess.
Now I cleaned it up a bit and turned it into a non-blocking OTP server application, but it is still just a proof of concept ...