The test suite

The language is tested by .pcpp files that describe themselves. testbed.exe finds them, builds and runs each one on every target and optimize level, and checks the result. Change the language, add your tests beside the official ones, and run the whole suite to prove you broke nothing.

Where the tests live

The suite is the folder bin\res\tests. Every test is one .pcpp file in an area folder:

bin\res\tests\
  lex\       the lexical rules: literals, comments, splices
  mod\       modules, imports, initialize / finalize
  dir\       the @ directives
  type\      types, records, objects, sets, choices
  decl\      declarations: const, var, routines, externals
  stmt\      statements
  expr\      expressions and operators
  intr\      the intrinsics
  test\      unit-test blocks and asserts
  cpp\       C++ mixing
  exe\ dll\ lib\   the three build kinds
  std\       the standard library units
  lsp\ dbg\  the language server and the debugger
  suite\     the test system's own tests

• A test's id is its folder path under bin\res\tests, with . between the parts, then its file name without .pcpp: type\type_record.pcpp is type.type_record, and type\record\type_record_align.pcpp is type.record.type_record_align.
• Sub-folders are for grouping only: put a large area's tests in one sub-folder per concern, type\record\, type\object\, and the id carries the path.
• A file's name (its stem) is unique in the whole suite, whatever folder it is in: two type_record.pcpp files are an error at discovery. Deps name tests by stem.
• Name a test <area>_<feature>.pcpp. A test of one mistake is <area>_<feature>_err_<what>.pcpp.
• A folder named fixtures (any case, any depth) is never searched for tests. A test's inputs go there: units it imports, C and C++ sources, a stdin file, an expected-output file. A .pcpp in a fixtures folder is never run as a test.
• A .pcpp with no spec block is not a test. It is listed as unspecced and never fails a run.
• Every file carries the project's file header comment, like every other source file.

The spec block

A test describes itself in one block, anywhere in the file, inside a /* */ comment:

/*
  @@test
    kind run
    expect
    | hello 42
    | [Heap] Allocs: 0, Frees: 0, Leaked: 0
  @@end
*/

• @@test opens the spec and @@end closes it. A file has one spec.
• The spec must sit in a /* */ comment that is compiled: not in a // comment, not in a region a conditional directive switches off, not in a cppstart block.
• Keys and values are separated by spaces and line breaks, in any layout.
• -- text is a note, to the end of its line.
• A value with spaces or quotes is quoted: "a b". Inside quotes, \" is a quote and \\ a backslash; nothing else is an escape.
• | text is one expected line of output. Trailing blanks are trimmed.
• The scanner is strict. An unknown key or value, a key given twice, a missing @@end, or a key the kind does not take is a discovery error, reported as file(line,col) with a code TS0001 to TS0017. A suite with a discovery error runs nothing.

Kinds

kind is required. It says what the suite does with the file:

• A manual file is one only a person can check: what the editor shows (the lsp_* files) or what the debugger does (the dbg_* files). The suite finds and lists it but never runs it, even when it is named alone.
• A tests file turns on @unitTestMode on; itself and holds test blocks (see Unit testing). A file with no tests fails: a vacuous pass would hide a broken file.
• A cli test checks pcpp's own command line. It runs once, on this machine, at x86_64-windows-gnu, debug, console.

Keys

• With no targets, a test runs on both official targets. With no subsystem, it is console.
• With no opt, a test runs at the optimize level the run asks for with -opt, and at debug when the run names none. Give opt only when the test is about one level, and it runs at those levels only.
• A value list takes each value once: targets x86_64-windows-gnu x86_64-windows-gnu is an error.
• stdin and expectfile paths are relative to the test file's folder: stdin "fixtures/my_input.txt". A missing file is an error at discovery.
• Output that differs by target takes one expect per target: expect x86_64-linux-gnu and its lines, beside a plain expect for every other target. expectfile takes a target the same way.

Expected output

• The program's standard output and standard error are compared together, in the order written.
• Line ends are compared as LF. Trailing blanks on a line, and empty lines at the end, are dropped on both sides. Everything else must match exactly. A failure shows the first line that differs.
• <DIR> in an expect line stands for the test file's folder, written with /. It is how a test expects a path that depends on where the suite is installed. It is not replaced inside an expectfile.
• <OUT> in an expect line stands for the suite's output folder, written with / (a cli test's pcpp names what it builds there: | Output: <OUT>/bin/x.exe). It is not replaced inside an expectfile.
• A cli test's expected output is all of pcpp's output, its build lines included. Colours, a window title a program leaves, and pcpp's Building <name>... <N>s progress lines (their number depends on the machine) are taken out before the comparison.
• A program runs in its own output folder, with the spec's args. With no stdin its standard input is empty.
• In a debug build the runtime prints its heap report when the program ends, [Heap] Allocs: N, Frees: N, Leaked: N, so write it where it appears, usually the last expected line (a DLL's report comes when the DLL unloads, before the host's later lines). Other optimize levels do not print it, and the suite leaves every [Heap] line out of the expected output when it runs the test at those levels: one expectation serves every level.

Report markers

A test that expects a diagnostic says so on the line the diagnostic points at, with a marker in a comment:

  GCount: int32; // ERROR SM0004@3: is already declared in this scope

• The form is SEVERITY CODE@COLUMN: text. The severity is HINT, WARN, ERROR or FATAL. The code is the diagnostic's code. @COLUMN and : text are optional; when given, the column must match, and the text must be the start of the message.
• A marker stands on the line of its comment, and one comment may hold several markers.
• A test's reports must match its markers exactly: every report needs a marker, and every marker needs its report. clang's reports count the same as the compiler's.
• An unmarked hint or warning fails the test, unless the spec says warnings allow.
• An ERROR or FATAL marker is allowed in an error test only. An error test needs at least one marker. Every other kind must build.
• A line no // comment can end (one that ends in a line splice, or a # line) takes its marker in a block comment whose text begins with //: /* // WARN LX0201@44 */. A block comment that does not begin with // is never read as a marker, so a description can show a marker without being one.
• One mistake per error test: a file with many mistakes tests the compiler's recovery, and only recovery tests do that on purpose.

Dependencies

A test that links a library or a DLL the suite must build first names it in deps:

  @@test
    kind run
    deps my_scale_lib
    expect
    | 6 x 7 = 42
  @@end

• The library is its own file, my_scale_lib.pcpp, a lib or dll module whose spec says kind dep.
• Before each run of the consumer, the suite builds its deps in that run's own target and optimize level, in dependency order, so the library is always the one built for the cell being tested.
• Every build writes into bin\output: programs in output\bin, libraries in output\lib. A consumer finds its dep with @addlibrarypath "$P:output/lib"; and @linklibrary "my_scale_lib";.
• Selecting a test selects its deps with it.
• Discovery checks the graph: a dep that does not exist, a cycle (the whole chain is shown), a dep that cannot build the consumer's target or level, and a dep test no test names are errors.

The live examples

bin\res\tests\suite holds one working test of each kind. They run with every suite run, so they are always current. Copy the one closest to your test:

The suite's own reports

The testbed's own codes are TS, reported at the file and line they are about. Any error found while discovering the tests stops the run before anything runs: Test discovery found <n> error(s): nothing was run. testbed -list shows them without running anything.

Running the suite

Run testbed.exe from the bin folder:

testbed -all                                   every test, every cell (each target at every optimize level)
testbed type.type_record                       one test
testbed "type.*"                               every test of an area
testbed "type.type_record*"                    a glob over ids
testbed -from expr.expr_a -to expr.expr_m      a range, in id order
testbed -all -t x86_64-windows-gnu             one target
testbed -all -opt fast                         one optimize level (repeat -opt for more)
testbed -all -kind error                       one kind
testbed -failed                                only what failed in the last run
testbed -list "lex.*"                          list the tests (-v: every cell)
testbed -root C:\my\tests -all                 only the tests under one folder
testbed -rpc status                            the running testbed's progress, as JSON
testbed -rpc stop                              stop the running testbed cleanly
testbed -rpc wait 600                          wait for it to finish (its exit code)
testbed -h                                     every option

• The exit code is 0 when everything passed, 1 when something failed, 2 for bad arguments, and 3 when the run was stopped. -rpc adds 4 when no testbed is running and 5 when -rpc wait runs out of time.
• A run goes cell by cell: every test on Windows at debug, then at safe, fast and small, then the same on Linux, so each target's toolchain and build cache stay warm instead of every test switching targets.
• One testbed runs per output folder: a second one started while the first runs is refused with TS0111. When the -rpc channel cannot be opened the run goes on without it (the warning TS0112), and when the stop file cannot be used the key still stops the run (the warning TS0110).
• A running testbed answers testbed -rpc <command> from another console: status (the cell, the test, the counts and how long since it last checked in, as one JSON line), failures (the failed tests so far, as JSON), stop (the same as the stop key: the running test finishes, nothing more starts), abort (the same as pressing it twice) and wait [seconds] (until the run ends; exits with the run's own exit code).
• To stop a run cleanly, press Ctrl+Shift+Alt+X (it works from any window, so it also stops a run with no console) or create the file bin\output\testbed.stop. The test that is running finishes, nothing more starts, and the summary says STOPPED after N of M. Press again (or create the file again) to end the running test at once: its processes are ended and it is listed as skipped, stopped. Ctrl+C is not a clean stop.
• A run prints one line per test and cell, and the first difference of each failure, then the totals. The last run's results are kept in bin\output\testsuite-results.txt, which -failed reads.
• Linux tests run through WSL. On a machine without WSL they are skipped with the reason, never counted as passed.
• Builds share one output folder, so tests build one at a time.
• A glob selects ids, so it starts with the area: "mod.mod_kinds*". A bare "mod_kinds*" matches nothing.
• -list takes the same filters as a run and shows the cells that run would use: testbed -list -v -opt fast "lex.*" lists each lex test at fast. A test the filters leave with no cell is not listed.

Validating a language change

1. Write the feature's test: one run test that uses the feature and prints what proves it worked, and one error test per mistake the compiler must report for it.
2. Run them alone until they pass: testbed mod.mod_my_feature mod.mod_my_feature_err_twice.
3. Run your area on both targets: testbed "mod.*".
4. Run the whole suite, one target and optimize level at a time. The toolchain caches its work per target and level, so this order reuses the cache for every test of a pass:

testbed -all -t x86_64-windows-gnu -opt debug
testbed -all -t x86_64-windows-gnu -opt fast
testbed -all -t x86_64-windows-gnu -opt small
testbed -all -t x86_64-windows-gnu -opt safe
testbed -all -t x86_64-linux-gnu -opt debug
testbed -all -t x86_64-linux-gnu -opt fast
testbed -all -t x86_64-linux-gnu -opt small
testbed -all -t x86_64-linux-gnu -opt safe

1. Fix what fails, then rerun just that with testbed -failed.

• A failing test is a defect to fix, in the compiler or in the test, never a check to weaken.
• A change that is not seen in pcpp's output, exit code or diagnostics is not tested yet: make the test print it.