Unit testing

Tests are part of the language: test blocks in any module, a runner an exe runs them with under @unitTestMode on, and twelve asserts.

Test blocks

A test block is test "name", at most one var block, then begin ... end;. Test blocks follow the module's terminating end.; one before it is PR0019.

module unit testing_unit;

exported routine Square(AN: int32): int32;
begin
  return AN * AN;
end;

initialize
begin
  println("unit init");
end;

finalize
begin
  println("unit final");
end;

end.

test "unit squares"
begin
  testAssertEqualInt(9, Square(3));
end;

module exe testing_runner;

@unitTestMode on;

import testing_unit;

var
  GHits: int32 = 0;
  GText: string = "a";

initialize
begin
  println("main init");
end;

finalize
begin
  println("main final");
end;

begin
  println("the body does not run in test mode");
end.

test "asserts that hold"
var
  LP: ptr to int32;
  LW: uint32;
  LI: int32;
begin
  LP := address of GHits;
  LW := 7;
  LI := 2;
  testAssertNotNil(LP);
  testAssertEqualInt(16, testing_unit.Square(4));
  testAssertEqualInt(7, LW);
  testAssertEqualFloat(2.0, LI);
  testAssertEqualFloat(0.3, 0.1 + 0.2);
  testAssertEqualStr("ab", GText + "b");
end;

test "asserts that fail"
begin
  testAssertEqualInt(10, testing_unit.Square(3));
  testFail("still runs after a failure");
end;

test "a raise ends the test"
begin
  raisecode(7, "stop");
  testFail("never reached");
end;

test "a fault ends the test"
var
  LQ: ptr to int32;
begin
  LQ := nil;
  LQ^ := 1;
  testFail("never reached");
end;

test "returns early"
begin
  GHits := GHits + 1;
  return;
  testFail("never reached");
end;

• Only one var block may come before a test's begin: a const block or a second var block is PR0025.
• A test sees its module's routines, variables and imports, and module state carries from one test to the next: a global a test changes stays changed.
• A test name is not empty (SM0111) and is unique within its module (SM0112); tests of different modules may share a name.
• return; leaves a test early; the test passes unless an assert before it failed.
• Every module kind may hold test blocks. A unit's tests run through an exe that imports it; the test blocks of a main dll or lib never run, which is the warning SM0113, and the module still builds.
• Test blocks are always checked, in or out of test mode: an assert given a wrong argument is an error even when the tests are not compiled in.

Test mode

@unitTestMode on; or off; stands in the main module only. Any other value is SM0100, and the directive in a dll or lib is SM0110: no program runs those tests.

• off, or no directive: the test blocks are checked but not compiled in, and the exe's body runs as usual.
• on in an exe: every imported unit's initialize runs, then the exe's own; then the runner runs the tests of every imported unit, then the exe's own, each module's in source order. The exe's begin ... end body does not run. Then finalize runs in the reverse order.
• The exit code is the number of failed tests, at most 255, the same on every target; the report's Results: line holds the real count.

The runner's report

unit init
main init

╔══════════════════════════════════════════════════════════════╗
║                     Unit Test Runner                         ║
╚══════════════════════════════════════════════════════════════╝

Running 6 test(s)...

✅ PASS: unit squares
✅ PASS: asserts that hold
❌ FAIL: asserts that fail
  🔴 …/samples/testing_runner.pcpp:91: testAssertEqualInt failed: expected 10, got 9
  🔴 …/samples/testing_runner.pcpp:92: testFail: still runs after a failure
❌ FAIL: a raise ends the test
  🔴 …/samples/testing_runner.pcpp:95: test raised exception 7: stop
❌ FAIL: a fault ends the test
  🔴 …/samples/testing_runner.pcpp:101: test raised exception 3: Access violation
✅ PASS: returns early

════════════════════════════════════════════════════════════════
Results: 3 passed, 3 failed, 6 total
════════════════════════════════════════════════════════════════
main final
unit final
[Heap] Allocs: 0, Frees: 0, Leaked: 0

• The runner prints a boxed Unit Test Runner title, Running N test(s)..., then one line per test, ✅ PASS: <name> or ❌ FAIL: <name>, a failed one followed by one 🔴 <file>:<line>: <message> line per failure, then Results: P passed, F failed, N total between two rules. <file> is the full path of the .pcpp file with forward slashes.
• A failed assert records its line and the test goes on: every failure of a test is listed, in order.
• A raise inside a test, or a fault such as writing through a nil pointer, ends that test as a failure reported at the test's own test line: test raised exception <code>: <message>. The runner goes on with the next test.
• In a debug build the heap report follows the finalize output.

Asserts

Twelve asserts, each a statement (no value), with names in their mixed case.

• The two-argument asserts take the expected value first, then the actual one.
• Each argument converts implicitly to its parameter's type (a uint32 into testAssertEqualInt, an int32 into testAssertEqualFloat); an argument that does not, or a wrong argument count, is SM0409.
• testAssertEqualFloat holds when the two differ by at most 1e-9 times the expected value's magnitude, or by at most 1e-9 when the expected value is 0: 0.1 + 0.2 equals 0.3, 1.5000001 does not equal 1.5.

Asserts outside a test

An assert in a routine or a body that holds does nothing. One that fails prints <file>:<line>: <message> to stderr, after everything the program printed before it, and ends the program with exit code 1 on every target; nothing after it runs (no finalize, no heap report).

module exe testing_outside;

routine Half(AN: int32): int32;
begin
  testAssert(AN mod 2 = 0);
  return AN div 2;
end;

finalize
begin
  println("finalize never runs after a failed assert");
end;

begin
  println("{}", Half(8));
  println("{}", Half(7));
  println("never printed");
end.

4
…/samples/testing_outside.pcpp:28: testAssert failed