Fri Feb 15 11:42:19 GMT 2008 bought a PPC Mac. Attacked webphoto again and made some progress, but photofs (the exif expander) needs a rework before its finished - the design had to change again; I never realised HTML was quite so badly specified. ------ Been thinking about cryptfs again. There are two seperate problems. encrypting filenames encrypting data As we are using CBC mode we want to ensure filenames with the same prefix don't have the same encrypted prefix, so we add a salt of a known length to the start of the filename and pad it with random data. The salt can be random bytes or even the checksum (adler being my favorite) of the name. The filename is then replaced by the base64 (with '/' replaced by some other chatacter) of the encrypted filename, however this lengthens the filename which may cause probelsm if the native filesystem holding the file has restrictions on filename length. For example you may run cryptfs on top of a u9fs session onto an MSDOS box with 8.3 filenames - not likely but it could happen; of course adding a salt further exacabates this problem. Alternatively filenames could be stored in the start of the data section of the file and the filename replaced by a short random ASCII name. Encrypting the data is not too hard though some aditional entropy would me nice to make the system more secure - some more random data. I could use the filename but that would mean that every time the filename changed I would have to re-encrypt the file. I could use the salt from the filename but I would like to have somthing reasonably big - 16 bytes feels good (wild guess I admit) and that would trigger the problems above. I could prepend the data file with a block of random data which is the salt for the filename and for the file data. this feels like a good solution but does grow the file a bit. The file should be encrypted block-by-block and reading or writing any part of the file means I must read or write a whole block. I can also use the block index in the file to ensure that two blocks with the same data (e.g. a file full of zeros) will encrypt to different data. The problem comes if two people are accessing the same data via two seperate invocations of cryptfs then we have a race for a given block of an encrypted file. If the file is held on plan9 I can set the exclusive bit on the file to provide some locking but it could be a remote ftp server via ftpfs, I can make no assumptions. If the host server is plan9 I could use Qid.version to determine if two users are accessing the same file and thus invalidating our cache though this would be a stat() on verey read & write and it is a very coarse indicator - the blocks may well not overlap. I could use RC4 which is a stream cypher and can encrypt a single byte at a time, thus there is no need for a read-modify-write cycle on a block and so no race. I had planed to offer a selection of algorithms: aes, des, triple-des etc. There are some other stream cyphers but again I am straying from known territory and I am worried of weakening the security of cryptfs, I would be happiest using a tried and tested cypher. I do know that RC4 can be weak if the first Kbytes of the stream are used (see the problems WEP has had) but provided this first part is scrapped I believe it is pretty strong. Tue Nov 13 16:12:46 GMT 2007 Some success writing a spelling corrector, now in my contrib - suggest.c.bz2 use it as spell document | suggest It permutes the word by alternatly: deleting a letter, substuting a letter by any other, transposing a pair of letters inserting any letter in any position. This gives it a set of possibilities. It then generates a soundex value for these words and compares that to the soundexes of all the words in the dictionary. It prints the first few words found which have a short edit distance (i.e. the number of deletions, insertions, transpositions, or substutions) from the word we started with. This is basicially what GNU aspell does (it stores the metaphone dictionary in a dbm database, I calculate them every time), and produces a useful tool which is quick enough to be useful on modern hardware. I added another level of validation to reduce disk access, rejecting modifications that produce invalid triagrams based on those I found in the dictionary. I was hoping this would speed up the program huguely - sadly it gave a 20% win which just about pays for itself. Tue Oct 10 12:19:39 GMT 2007 Had a play at generating photo gallery automaticially from a directory of thumbnail jpegs. The important point for me is to require no configuration files and no initial "generate small images" script. modern digital cameras store quite a bit of metadata in their jpegs (the EXIF standard), and this usually includes a thumbnail image builtin. spent an unreasonable amount of time understanding html to get this to work, and I should know more really - sure the HTML style police will be after me. Keeping the pages very simple (clean in my terms) - struggled to find some nice navigation arrows. I got some from gimp which are OK but hopefully I will get some better ones later. Wed Sep 19 12:59:46 BST 2007 9win progress stopped children eat all your time! just some other thoughts, would be good to have a plot(1) driver that generated a plan9 bitmap rather than only the one that talks to /dev/draw. I know you can generate a bitmap from your window but this becomes much more difficult if you want to generate graphs offline (via cron) for inclusion in web pages. should be very easy really, in fact there may be somthing similar on comp.sources.unix or the like. Also I would love to write a samterm replacement which opens two or three files and does a two-way or three-way merge between them (by three-way I mean the two files have a known common ancestor). This would be a GUI app which would allow you to take the text from either of two windows to be fed to the output or edit the output by hand. sort of a GUI idiff(1) really. Tue Aug 14 14:35:15 BST 2007 Discovered to my amazment that Bruce Ellis is working on somthing like 9win (he calls it 9ee) also. I hope we can share some work. I always planned to pick up the RangBoom windows IFS driver to allow all windows programs to access the 9win/9ee files I create - IE both microsoft word and 9win's ls(1) will see /dev/cons /proc /fd /net /cron etc. This is improtant as the biggest argument against Russ Cox's 9apm code was the mental shift nescessary when moving between windows apps and plan9 ones. Tue Jul 10 14:14:00 BST 2007 Busy time of it lately, beacme a father which take up more time that I though possible, also very busy at work releasing products. 9win is progressing, libc is now pretty much complete, all utf8 compatible and uses no microsoft C runtime DLLs, only direct windows API calls, I only need to finish exec() and rfork(). Exec is not too bad once you accept that you will have to run a thread to tidy up up orphans which are not going to be waited for, and also to relay on Notes for the real child - as the plan9 pid (Windows Tid) will change when the process execs. There is also the fun of quoting for the execed command line args but this has been done before. rfork is more painful. Most flags can be simply implemented but rfork(RFPROC) is mode difficult. The cygwin way is to CreateProcess() of yourself with the new process suspended and then modify its environment to be a mirror of your own; This is a good (abet slow) fork emulation. The book Windows 2000 Native API Reference has an intriguing snippet which almost does a native fork, unfortulately cloning our .data segment also appears to clone the CSRSS's initialised state and so fat the only way to force it to reconnect is to call an unpublished entrypoint (ugh!). a third alternative I have seen is to pretend we are doing a vfork() and switch internal context to the child process and return. Then on exec() exit(), or abort() we longjmp back to the saved parents thread. This assumes we only use rfork(RFPROC) as a prelude to exec() which may or may not be good enough for the plan9 apps... rfork(RFPROC) is also a bit knotty, I think I will just us a thread, copying the stack of the parent. This should work OK, though on return from the fork the app must not make assumptions about the addresses of automatics. This is very kludgy but I think I may get away with it given how most plan9 code is written, but I will have to experiment to find out. ---------------- Hackers again, I just got anoyed and blocked all ftp access from china in my firewall and the problems pretty much went away, this is a real shame but I don't want to waste time with these people. Fri May 18 11:32:15 BST 2007 did a bit more work on ps, reworked it to print some more useful info even if you don't have SE_DEBUG privilege. I also dropped win16 support (we don't really use it any more) and added an executable-type field in ps-s usual state field. had another hack at 9win - getting closer, main stumbling block seems to be crt0 which I can pinch/hack from mingw for now (lcc might also have some good ideas). Thanks to russ for a gcc-ised libc/port/pool.c free of kenc-isms. Wed May 9 11:01:42 BST 2007 had a happy hacker trying to bruteforce my ftp account, no real danger of them succeeding as they are using the account name Administrator. Interestingly if you google for their IP address it is listed on a couple of andt-brute-force IP lists. I modified ftpd to add an exponentially increasing delay on failed passwords and limit the number of attempts allowed. I also included a delay on the startup time of ftpd based on the number of failed sessions from that IP address. I submitted this as a patch but geoff quite rightly chopped it down - I was tirred and got over excited. I may have a go at adding similar functionality but in the /rc/bin/service scripts, based on a new exit starus string from telnetd, sshserve, and ftpd. Wed May 2 10:16:20 BST 2007 Had a play at running Plan9 (386) binaries unchanged under Windows. Not as difficult as it sounds, you can memory map (CreateFileMapping()) the executable file, MapViewOfFileEx() the text as executable (FILE_MAP_EXECUTE) and the data and bss as copy-on-write (FILE_MAP_COPY). The stack you can VirtualAllocEx() to the aproopriate address. Windows executables are relocated on load, but you can force them to a known (fixed) address with linker options so the emulator could force itself to a location outside plan9 normal executable's space. Then you can use SetUnhandledExceptionFilter() to catch the "trap 64"s that reprisent plan9 system calls (and there are only 50 odd of those to emulate). I can even put a placeholder segement under and over the stack and and after the heap so I can trap some memory access bugs. Unfortunately I now discovered that the pagesize on Plan9 x86 is 4kbytes whilst on Windows it 64k. This means that the text, data/bss/heap, and stack must start on 64k boundries, whereas in plan9 executables they are placed on 4k boundries. I can get round this easily enough using 8l(1)s -R option to specify a different alignment (rounding) for the data segment but that means relinking all plan9 executables for windows, which is what I was trying not to do. Also, I was very suspicious that no-one had tried to do this for Linux or MacOs binaries running under windows. why hasn't anyone tried this before? maybe its more difficult that I thought? So this project is a dead end, shame really. Fri Mar 23 13:14:48 GMT 2007 Distracted with trying to bugvix cifs and aquarela, with some success but it is timeconsuming work, espicially the latter as I don't even know the code. Also looking at 9pm, trying to see how I could build windows plan9ports like environment. At present I think it would be best to build a central plan9-like kernel with a mount driver, devfs, devcons etc (like Russ's example and drawterm) rather than emulate at the level of libc like 9pm did; Its not a clear decision though. I am starting by hacking drawterm which appears to be a very clean environment I.E. only minimaly changed from the plan9 kernel. Tue Mar 13 08:35:35 GMT 2007 I found an example of an animated gif which demonstrates the bug: hget http://www.quintile.net/broken-animated-image.gif | gif The problem appears to be that the image is incorrectly being erased between displaying each page of the animated sequence. Fri Jan 26 11:06:18 GMT 2007 Very nice detail about how Google Maps works, http://www.codeproject.com/useritems/googlemap.asp it would be fun to write a client for Plan9 if I only had the time. Been playing with the Win32 API learning a bit more, u9cpu is getting a little closer. Previous work ~~~~~~~~~~~~~ Wed Jan 17 23:13:17 GMT 2007 u9cpu a CPU server for Win32 would make my life quite a bit easier, this is not too difficult if I port the TLS code from p9p (or perhaps the code in secstore). The pain comes in supporting Windows and to allow the server to automount network drives (hopefully by examining /mnt/term/dev/ns to discover which are in use). I also need to ad another authentication method which passes the users username and password for re-authentication on windows (sad but true). I may be able to get the windows mount driver code from friends which would be great but they have comercial interests so it may not be possible. cifs DFS implementation is a bodge, it only works for a single server with dfs redirections between its shares (what we have) - It is NOT able to connect to a seccond machine. It should be implemented by binding a different part of cifs's own namespace or by starting another cifs instance (if the redirection is to a remote machine). MAC signing doesn't work. All the code is there but the signatures are wrong. I thought this was required for Win2k3 but it turns out MAC signing is optional so it is not a requirement. As part of the work to get MAC signing going the factotum keys are user/pass rather than MSCHAP/MSCHAPv2; the mschap auth being done in the cifs app. This is a temporary measure. When MAC signing works the MSCHAP and MSCHAPv2 fields passed between cifs and factotum (and pop and IMAP I think) will have to be modified to add the MAC signing key which is not present currently. Shares are enumerated using RAP (Remote Adminstartion Protocol). This has quite a few restrictions, most notably that share names cannot be more than 13 chars long. The proper solution to this is to use the MS-RPC (Microsoft Remote Procedure Call) interface. This has many more admin features and no 13 character restriction on share names. I have not started MS-RPC and have a fear that it may require SPNEGO auth or perhaps kerberos, I can find no definitive answer as yet. No support for changing passwords exists at present, a nice simple synthetic file into which you write your old password and new password and read back either a blank or an error message should be enough. This could be implemenetd with RAP and should not be a problem. I now have password ageing enabled at work (sigh) so my need for this is increasing so I may implement it soon (Dec 06). I feel guilty about adding all the extra synthetic files for Users and Shares etc into the main cifs heirarchy. These should really go in a parallel heirarchy which could be mounted using a different attach specifier, somthing like "admin". cifs could abstract out sessions completely so the attach specifier could describe the host to attach to and the username to use. Thus dfs would become simply an auto-mount or an auto-bind. setting/resetting rdonly flag effects mtime - it shouldn't. rap and rcp could (should) be seperate programs which generate their own synthetic filesystems based on messeges read/written to a synthetic file in cifs's filesystem; the protocols layer quite cleanly so why not the filesystem implementations? More info on RCP here: http://www.hsc.fr/ressources/articles/win_net_srv http://www.xfocus.net/projects/Xcon/2003/Xcon2003_kkqq.pdf ncpfs All development work stopped, my last NetWare server has gone. Ncpfs works fine but doesn't support NDS, only Bindary access so you must attach to each server you want to mount individually rather than being able to attach to an NDS Tree like you can from windows. One recent change - tiny edit as ``reject'' is now a keyword in our compilers (C99 I guess). cvsfs No outstanding problems (well none that can be fixed given the CVS protocol). There is a possibility of making it a bit quicker to connect by using aditional features of NTcvs (NTcvs is not limited to Windows and implements an extended feature-set). I haven't looked into this cryptfs Mostly done but not finished as I couldn't decide how I wanted it to work. Russ produced a set of patches for kfs(1) to encrypt data (see /n/sources/contrib/rsc/cryptfs) but i intended cryptfs to be a pass-through file server encrypting data and filenames on a remote file-server. The pass-thru file server is done and support for AES, DES, DES3, and IDEA (IDEA lifted from inferno), however it stalled there. pptfs done, works, bit clunky as the page numbers seem to be a bit random (no idea why) and somtimes unnescessary gubbins appears in some of the text files. pptfs cracks the powerpoint stream which itself is cracked from the OLE FAT filesystem by aux/olefs (part of the plan9 distribution). basicially you want to do: aux/olefs mypresentation.ppt pptfs cat /mnt/doc/ppt/*/*.txt page /mnt/doc/ppt/*/*jpg* /mnt/doc/ppt/*/*png* ndb/dns I submitted a patch which attempts to prevent dns poisioning which was sorry'ed as it was incomplete, I should have another go; see /n/sources/patch/sorry/dnstcp-norecursion for more info. Still an outstanding bug where looking up an MX record on an non-cached DNS record gives "failed" rather than "resource not known", this is easily provable as doing a lookup of the A record on the name (to force it into the cache) results in MX lookups returning "resource not known" correctly. This is only seen when using upas/smtp with SMTP gateways which don't have MX records but which are passed on the "-h host" option. If "-g gateway" is used then upas/smtp works reliably. sqlfs Started but haven't got very far, look at odbc.c in inferno for an apropriate file interface; it would be very nice if sqlfs was compatible with this. chatfs Got code snippets and docs to support for MSIM, TOC(yahoo), and IRC (thanks Russ). Basic framework done and works for IRC (I use it to chat), it needs the other drivers added and the file interface done Also, would be nice to attach a faces-like client showing who has been speaking in the last (say) 10 mins Bugs and "missing" features ~~~~~~~~~~~~~~~~~~~~~~~~~~~ hget & webfs should support: digest auth and proxy auth compressed (gzip) data transfers caching (bought the book...) upas/fs add a "realname" synthetic file for each message. This would make it easy to run a script on a message to pull the users picture from ldap and feed it through mug(1) and write it to $home/lib/face mug(1) should understand colour - maybe it is replaced by the image editing tool below. marshal bind the current mail item over /mail/fs/mbox/current so scripts can find it easily bug in pr when printing two collums some lines in the first collum get lost, dependant on their length - needs tested again. 9grid encrypted link between auth servers, using public keys, or replace p9sk1 with p9pk1 (public key auth for 9p) resource limits: core memory limitation I/O operations limitation fair share scheduler controlled by /lib/ndb/auth network bandwidth restriction controlled by /lib/ndb/auth disk quotas ??!?!? I wonder if these could be done by interspersing a fairly transparent filesystem beind the users rc(1), this could simply poll to see I/O use etc? proabaly not good enough but worth a punt I guess. I started to write a paper on cross domain authentication and the arguments for adding Public key authentication. This needs to be finished. tojpg(1) we really should have tojpg in this day and age, though perhaps topng(1) makes it irrelevant? png(1) png(1) cannot read some of the images created by topng(1), this is very sad. gif(1) animation doesnt work properly with optimised gifs (where previous frame is susposed to show through depending on Alpha blending). Vaopur ware - it would be nice but I'al probably never get round to it... ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ image processing GUI A simple program that displays an image like page but allows you to select a reigion of interest (ROI) and push that through a list of filters. The list of filters should have a macro language to describe them and their mandatory args, this list should live in /lib/imged.rules or the like. Thus local tools could be added. there would also need to be some standard widgets like the pick-a-colour widget which would be envoked if the tool you wanted needed it, e.g. # /lib/imged.rules #name cmdline... # crop crop -b $colour -r $rect # this would cause the chose-a-rect and pick-a-colour widgets to be envoked. # resample resample -x $percent % # here we just resample the image with a percentage change we also need a load more tools, glg (gamma, lift, gain) - see mug(1) for a very elegant way to offer these parameters. These should also be available in red, green, blue or cyan, magenta, yellow space. not sure if this model will work (toolkit of external commands which when envoked request a widget to get their parameters) but at least its clean. There is loads of interesting tools which are quite simple to do gain lift gamma median-filter noise reduction rotate fast for 90° increments, 3 shear algo for degrees histeq auto gain/luminance fft2d windowed 2D FFT project image by a field of vectors enhance hf gain/loss, linear chrispen nonlinear sharpening etc. Video processing toolkit A little language which would allow you to process a stream of video by pipeing it through commands and also using low level (pixel level) operators. This should look as much like a proceedural language as possible but be able to be compiled into a set of dataflow operations and perhaps the pixel level ops should be JIT compiled to machine code. The important features are high efficency, low latency, a toolkit approach like the image editor above, and video streams are arbitarily long, they are not sequences of images, it is video. the system must respect interlace, 3:2, 4:5, and other strange field cadences data should be passed to child programs via pipes but the header of the file may contain a shared memory segement address at which the image data may be simply read. see: Eric Grosse, "How shall we connect our software tools" video must be held in files, the header of which contains simple, ascii token=value assignments, these are terminated by a blank line. pixel streams which are sampled in the same matrix may be interleaved but those sampled differently must be put in seperate files, eg: R,G,B can be held in a single file, as can H,S,L but Y must be seperate from U and V as they are sampled at half the rate of Y. copyright='Cable News Network' title='The JFK shooting - new evidence' lines=1080 # active lines per frame pixels=1920 # active pixels per line cadence=1:1 # ie video clock=74.25e6 # 74.25 Mhz sample clock chans=y # luma channel only aspect=16:6 # display on a 16:9 monitor colour=SMPTE274 # colourspace defined by SMPTE274 specs ... Note: there is NO specification of the length of the sequence, you just read until EOF (or forever if your input is /dev/tv). the chans are marked as 'y' only so the 'uv' channel must be in a seperate file. webdavfs bought the book, nothing more update X11 port make it work fast on all plan9 displays and make it offer 8, 16, and 32 bit pixmaps. fossil allow it to snap a vac score so files already absorbed can be amalgamated into venti's dump filesystem. libder.a a cannonical library for use in snmpfs, ldapfs, kerberos etc kerberos kerberos support in factotum, no pressing need, just it "should" be there. ldapfs usefull in a Windows enviroment to find peoples phone numbers/email addresses tsv Microsoft terminal services client - ISODE has a lot to answer for, the only documentation is some ISO docs and the source of the linux rdesktop program - see sourceforge.