.HTML "Shifs - a simple way to transform and build dynamic file servers .TL Shifs A simple way to transform and build dynamic file servers .AU Uriel M. uriel @ cat-v.org .AB .PP .I Shifs is .I Shifs is a way to transform and generate simple dynamic file systems that can be used directly from the command line and can take advantage of the standard tools used to manipulate text streams. .I Shifs descrived here is only a prototype but it is already functional enough to be useful and replace some specialized components. .AE \" .2C .NH Introduction .PP .I Shifs uses a very simple language to deffine a set of .I paterns that trigger .I actions . In that sense it is similar to .I awk but instead of handling text streams, it works with file trees. .I Shifs was born out of the desire to have a simple way to transform and exten the data expossed by file server in a generic and extensible way. Editing the source of existing file servers is too intrusive and makes it harder to maintain backwards compatibility for applications that might expect the old interface. One could avoid that in some cases by adding new synthetic files to the file server that would export the data in the desired format, but that cluters the file server with files that might have a very specialized purpose that is not related to the function of the file server. In other cases access to the file server might not be available, or one might want to transform a whole tree that might include files exported by different servers. .PP Other goal was to make it convenient to leverage the power of the standard application toolset of the operating system and in that way bring the .I software-tools philosphy to the world of dynamic file server creation, and do so in a secure and convenient way. .PP The more inmediate task that prompted this interest was the need to overhaul the HTML generation code in .I wikifs which consits of almost a thousand lines of C code used to first parse the internal text format of wiki pages and then generate the HTML output. Considering that the same task could be acomplished by a dozen lines of awk and other standard tools, it appeared worthwhile to take advantage of that approach without discarding the convenience of a file system interface. .NH suckfs .PP A first attempt was made to implement in C a whole language called .I suckfs (because it .i sucks files) in the spirit of awk where rules would match path names and actions would dynamically generate or transform the contents of the matched files. But soon it became clear that the language would need to be too complex to even perform the most simple tasks, and it failed to acomplish the main goal that was to take advantage of existing tools and languages without adding extra complexity. Also it was apreciated that it would be convenient if the facility could be available to as many platforms as possible. Inferno provided this and was a good oportunity to experiment with Limbo. In hindsight this was a very good choice as it simplified some problems considerably. In particular using the .I sh module made everything much easier. .NH The shifs language .PP The language was simplified to the bare basics, providing a fundation that could be extended in the future as more complex interactions become feasible. The language has a syntax somewhat similar to .I mk and consists of .I rules , each composed of a .I pattern and a set of .I actions . .NH 2 .PP A pattern starts at the begining of line and consists of an operation specifier ( .I r for read, .I w for write, etc.), followed by the regexp match operator .I ~ (other operators might be added in the future), the rest of the pattern up to the end of line is the argument to the regexp operator which is a regular expresion that is matched against file paths. .PP \" .1C .KF .P1 r ~[0-9]+/[a-z]+$ .P2 .ce .I "Figure 1. Sample shifs pattern" .KE .PP .NH 2 Actions .PP To complete a rule a set of .I action lines follow the pattern line, action lines should begin with a .I tab character and use the Inferno sh syntax. \" .1C .KF .P1 r ~^today$ date|sed 's/ .*//' .P2 .ce .I "Figure 2. A complete shifs rule" .KE .PP The environment in which actions are run includes some convenient variables set by shifs before the action is run, of special interest is .I $FPATH which contains the full path that matched this rule. .NH Execution .PP It is easy to run .I shifs from a terminal window, the arguments that the following format: .B shifs [ .B -po ] [ .B -r .I /source/tree ] [ .B -m .I /mount/point ] [ .I rules ] .PP The .I -p option allows to 'pass-thru' files that don't match any rule so they are retrived unchanged from the underlying file server. The .I -o makes the exported file system read only. The path passed to .I -r is used as starting point for the exported file tree, and actions use it as their working directory, if the .I -p options was used all the files under that tree that don't match any rule will also be accessible without any modification. If .I rules is not specified the rules will be read from stdin. .NH Walking, opening and reading. .PP .I Shifs keeps track of walk operations, when a .I fid is opened the full path is matched against each rule, if a rule matches a new process is spawn that sets its stdout to a pipe from where the subsequent read operations will feed and then executes the contents of the associated action. .NH Uses .PP Some of the tasks where shifs can be helpful are: performing simple transformations to the contents of arbitrary file trees, for example when importing a file system from a windows system one can easily tranform all files to use the Plan 9/Unix new line format by simply using .I tr(1) . Other useful task is to format data from its canonical format into formats convenient for export, a good example of this is adding html formating before exporting a file tree over httpd. Also it can be convenient to create synthetic files that collect and sumarice the contents of whole file trees. Finally one can even create completely synthetic file trees, for example here is a file server that returns the square of any given number: \" .1C .KF .P1 % shifs '~^[0-9]+$~ {load expr; echo ${expr $FNAME $FNAME x}}' % cat /mnt/shifs/4 16 % .P2 .ce .I "Figure 1. Sample shifs pattern" .KE .PP .NH Conclusions and future directions .PP The current version of .I shifs is still very much a prototype with many rough edges, but so far it has demonstrated to be a good tool to experiment with 'one off' file servers. And that is probably its greatest power, just like shells allow you to easily write small throw-away programs that when found useful can be codified either as shell scripts or in some cases rewriten in a " .I real" language. It also manages to at least partially suceed in its other goal of making the more traditional Unix/Plan 9 set of text manipulation tools more easily accessible when writing simple file servers. .I Shifs is still somewhat fragile, but still can be useful in experimenting with changes to the interface of existing file servers and in working as an adaptor between file system interfaces. The main areas that still need work is handling of write operations and transformation of directory listings. The ruleset language is probably somewhat simplistic and naive and might need some redesign. Perhaps for better integration with a Plan 9 environment a rewrite in C could be considered, but maybe it would be best to instead try to take advantage of the "native" Dis. .NH Aknowledgements .PP Awk is clearly the main source of inspiration, but the more general tool philosophy underlines the goals of the project. Also one must aknowledge how both Limbo and the sh module in Inferno made implementation almost trivial. .PP I am not aware at the moment of any similar projects, but it is very likely that someone has thought of a similar thng before, my apologies if I have not noticed it. .NH References .LP [Inferno] Inferno Manual page for Sh(1) .LP [Aho79] A. V. Aho and B. W. Kernighan and P. J. Weinberger ``AWK - A Pattern Scanning and Processing Language'', .LP [Kernighan76] B. W. Kernighan and P. J. Plauger ``Software Tools''