Writing Filesystems in Perl

Richard Dawe

<rich@phekda.gotadsl.co.uk>

http://homepages.nildram.co.uk/~phekda/richdawe/Birmingham.pm/

The Idea

Unix has a philosophy of ”everything is a file”

XPaths look a bit like filenames

What if we could mount an XML file as a filesystem?
mkdir -p /mnt/xml/html/head
echo ”Some random HTML document” \
> /mnt/xml/html/head/title
mkdir -p /mnt/xml/html/body
echo ”Some random HTML document” \
> /mnt/xml/html/body/h1

Filesystems in Linux (1 of 2)

File operations abstracted through Virtual File Switch (VFS) layer, e.g.:
int (*rename) (struct inode *, struct dentry *, struct inode *, struct dentry *);

VFS

Program

Syscalls

ext2

procfs

User space

Kernel space

Filesystems in Linux (2 of 2)

Filesystems can be loadable modules
=> load new filesystems on the fly

But writing kernel modules is hard; hard to debug:

crashes

deadlocks

race conditions

FUSE (1 of 2)

FUSE is ”Filesystem in Userspace”

Linux kernel module (fuse.ko) plus user-space library for writing filesystem daemons

Library API consists of callbacks for file operations. Similar parameters to C API. E.g.:
int (*rename) (const char *, const char *);

Language bindings: C, Perl, Python, Java, TCL

FUSE (2 of 2)

Kernel module forwards VFS operations to a user-space daemon

Daemon communicates with kernel via special device, /proc/fs/fuse/dev

VFS

Program

Syscalls

fuse

User space

Kernel space

filesystem

Read request

Write reply

Semantics of an XML filesystem

Don't support narrative documents (i.e.: HTML)

XML node with textual contents is a leaf node

Empty XML nodes or XML nodes with children are directories

Attributes map to extended attributes

Hardlinks/symlinks could be supported by XLinks?

Optionally enforce the DTD / schema

But how would it work?

<example><node/><node/></example>

How do you create a node under the second <node/>?

Solution: Suffix all nodes with an index, e.g. node@@0, node@@1.
mkdir /example/node@@1/node@@0
echo ”foo” > /example/node@@1/node@@1

The index of new nodes must be the next one, e.g.: node@@12 is not valid.

Live demo

Mount an XML file

Find

Cat a file

Create a directory

Create a file

Append to a file

Development experience

FUSE builds easily with Linux 2.6.10, 2.6.11.

Hard to debug, because there are three layers: kernel, library, Perl bindings

E.g.: Where do I look when open() syscall fails with an EIO error? Turned out to be inconsistency in xmlfs.

Extended attributes in Linux 2.6.11 and later, i.e.: cutting edge

Perl module lagging a bit – fixing that

Other FUSE filesystems

From the FUSE home page:

”FunFS is an advanced network file system with a simple goal: to be better than NFS.”

”EncFS provides an encrypted filesystem in user-space.”

sshfs – mount filesystems over ssh

fusedav – mount WebDAV shares

Someone using FUSE with > 1TB of data

Further Reading

FUSE home page <http://fuse.sourceforge.net/>, fuse-devel mailing list

CPAN <http://search.cpan.org/perldoc?FUSE> or FUSE CVS <http://cvs.sourceforge.net/viewcvs.py/fuse/> for latest Perl bindings

”Linux Kernel Development” by Robert Love – good explanation of VFS layer

xmlfs home page <http://homepages.nildram.co.uk/~phekda/richdawe/xmlfs/>

2005-03-30T14:28:21