Sunday, October 14, 2007

Mount Amazon S3 on your Mac

I recently discovered s3fs, a fuse based file system which allows to mount an Amazon S3 bucket like a normal file system at your PC. With other words, it is very easy and relatively cheap now to expand your local hard drive with terabytes of backup or Photo/Video storage. And with the recently announced Amazon S3 SLA, it seems Amazon is committed to continue with this service.
s3fs is currently only provided in form of source code. But with a little bit of tweaking I got it compiling and running on my Mac, see below the instructions and the required Mac specific modifications:

Get and install the latest MacFUSE-Core. This is just background process, without any
GUI elements.

Checkout the s3fs source code. Because it is just one file, you can even copy it manually in a newly created directory and add the following line to s3fs.cpp
(after #define FUSE_USE_VERSION 26):
 #define __off_t off_t
Start a terminal, change to the directory with the s3fs.cpp file inside and prepare the environment:
  export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig
Next modify the Makefile (or create one, if you just downloaded the s3fs.cpp):
  all:
g++ -Wall -D__FreeBSD__=10 -D_FILE_OFFSET_BITS=64 $(shell pkg-config fuse --cflags --libs) -lcurl -lcrypto $(shell xml2-config --cflags --libs) -ggdb s3fs.cpp -o s3fs
@echo ok!

clean:
rm -f s3fs s3fs.o
If everything went well, you should have now a binary file s3fs in that directory. Now create a file /etc/passwd-s3fs which contains just your Amazon ID and secret key separated by ":", e.g:
  example-id:example-secret-key
now you create a new Amazon S3 bucket. I have been using the S3 Browser for that. Define a mount point, I just created a new directory in my home folder for that. Now you can mount that directory to your newly created Amazon S3 Bucket by running the following command:
  ./s3fs your-bucket-name your-mount-point-directory
Now you should see the MacFUSE icon in the Mac OS X Finder. And any file you put into your mount point directory is now physically stored at your bucket at Amazon S3. It's not very user friendly yet, let's hope the MacFusion guys integrate it soon into their excellent Fuse tool. I plan to integrate this S3 bucket access via s3fs into my software development toolchain. Because everything is scripted there, s3fs works fine for me, even if lots of important features are still missing.

Update:

In case you don't have libxml2 installed already, you need to install it first:
  sudo port install libxml2

9 comments:

mj said...

Thanks for this article. I'm going to try it. I've been using a "Yellow Machine" server, but now I want something I can reach from anywhere. Dot mac should be able to do this, but it's too small and expensive.

Anonymous said...

The link to the one file (s3fs.cpp) is not good. Where can I get it at?

Roberto Saccon said...

I updated the link, but the project is in development , so things are changing and might change again, maybe better if you start at http://s3fs.googlecode.com to look for that code.

Stephan said...

i've got an error after executing g++:

-bash: shell: command not found
Undefined symbols:
"_main", referenced from:
start in crt1.10.5.o
ld: symbol(s) not found
collect2: ld returned 1 exit status

chris said...

I'm getting the same error as Stephan... Do you know what might be the problem Roberto?

same machine said...

I'm having the same problem as well. Has anyone figured this out?

Mad Hatter said...

For everyone getting command not found on g++: you need to install the compile tools. See this: http://aplawrence.com/MacOSX/xcode.html

dandante said...

I got it to build and sort of work....I can copy files there, though I get the following message:

cp: test.txt: could not copy extended attributes to /Users/me/s3mount/test.txt: Attribute not found

When I do an:
ls ~/s3mount
nothing shows up - not even files that I have placed in that bucket using s3sync.rb.

However, the file does appear in S3 Browser, along with ._test.txt, a 4K file (even though my test file is only 210 bytes long).

Finally, no icon shows up in Finder.

It seems like this software has a ways to go

vasilis said...

Hi

I am getting the following type of error messages when trying to mount:

The process has forked and you cannot use this CoreFoundation functionality safely. You MUST exec().

I have 10.5.6

Any ideas?