This test suite for nhc98 is far from comprehensive, and is nowhere near as clever as ghc's suite. But it is better than nothing. Running tests ------------- Use the provided script: ./runtests clean >/dev/null # clean up after a previous run ./runtests >test.log 2>test.err # run all the tests Stderr will contain a line for each failing test, telling you whether it failed at runtime or compile time. The most useful information is the summary of numbers passing/failing (see below). Stdout will contain a log of the complete test run, showing the compilation commands executed by hmake, and diffs between expected compiletime/runtime output/errors and the actual output/errors. Expectations ------------ At the time of writing (Aug 2004), there are 134 test cases. Unfortunately, we don't have a very good way to distinguishing which cases are intended to fail and which are intended to succeed. But the usual kind of result is something like this: 111 built ok / 23 builds failed / of which 23 known / 0 bad 110 outputs match / 1 outputs differ 110 errors match / 1 errors differ The best numbers for comparison are basically those on the right. We expect 23 build failures (23 known/0 bad), 1 output differing from expected, 1 error differing from expected. If nhc98 is built with hbc, then we get 24 build failures (1 bad). To add new tests ---------------- Each test occupies a separate directory containing (at least) a file called Main.hs. This allows multiple-module tests to be created easily. Choose a directory name that conveys something about what the test should accomplish, e.g. conformance98/layout/rightbrace. Aside from the Haskell code itself, you should provide a sample stdin, with corresponding stdout and stderr, in the files named *.stdin, *.stdout, *.stderr. (You may omit any of these if they would only be empty. Usually, the files are named in accordance with the testcase, e.g. rightbrace.stdin.) If the build is expected to fail, you should provide a *.hmakerr file for comparison.