Running the Tests

This guide answers "how do I run the tests I care about, right now". It is task-oriented. The harness internals and the traps that have burned us live elsewhere:

• test-suite-portability-guide.md -- portability and performance pitfalls (Bash 3.2, ASan, leak-check coverage, sharding parity, timeout budgets).
• test-runner-contract.md -- the contract for writing a test framework or harness.
• diagnosing-the-compiler-guide.md -- debugging the compiler once a test has told you something is wrong.

Two rules hold for every command below:

• Wrap every suite run in a 12-minute timeout (timeout 720 ..., or timeout: 720000 in the Bash tool). See CLAUDE.md.
• Run one suite at a time. Two suites, or a suite and a build, on the same machine produce failures that look exactly like product bugs. See CLAUDE.md, "Overlapping runs cause false FAILURES".

1. Which harness runs what

Each ctest target is one script. To see which script a target runs:

ctest --test-dir build -N -V -R '^tur_r7rs_gc$' | grep 'Test command'
# Test command: /usr/bin/bash "tests/run-r7rs-gc.sh"

Then run that script directly. It is faster than going through ctest, and its output is not truncated.

2. Narrowing a run

tests/run.sh

They compose. One fixture, from scratch:

TUR_FORCE=1 TUR_TEST_FILTER='^arith$' timeout 720 bash tests/run.sh

The output goes to stdout: a PASS/FAIL line per fixture, then summary: P passed, F failed. A failing fixture's actual.stdout and actual.stderr are left in its directory.

There is no TUR_FILTER. An unknown variable is silently ignored, and you get the full suite.

tests/run-turi.sh

TURI_FILTER='r7rs-srfi' timeout 720 bash tests/run-turi.sh

The summary reads P passed, F failed, S skipped of D discovered. The run fails if those do not add up.

tests/run-jit.sh

TUR_TEST_FILTER (or JIT_FILTER, which wins), TUR_TEST_SHARD, and TUR_TEST_LIST=1 (list only), as in tests/run.sh.

tests/run-markdown-spec.sh

The CommonMark and GFM spec examples through tur/markdown (ctest tur_markdown_spec). They live in a sibling checkout of turmeric-lang/commonmark-spec, which the script PASS-skips without. Clone it to ../commonmark-spec at the commit in tests/markdown-spec/PINNED, or point MD_SPEC_DIR at a checkout.

• Engines. MD_SPEC_ENGINE picks them: compiled, interpret, jit, both (compiled and interpret, the default) or all.
• What fails. Only a regression: an example in tests/markdown-spec/ratchet.txt that no longer passes on some engine.
• Arguments go to tests/markdown-spec/run.py:
  • --only c12,c13 runs those examples (c CommonMark, g GFM);
  • --show N prints input, expected and actual for the first N failures;
  • --update adds newly passing examples to the ratchet.

tests/run-markdown-diff.sh

tur/markdown's HTML against the reference implementations' (ctest tur_markdown_diff). It renders every Markdown file git tracks here, every README in ../turmeric-spices when that checkout is present, and random documents from fixed seeds, then makes two comparisons. Every document whose HTML differs fails the run.

• Plain CommonMark, against commonmark.js 0.31.2, the implementation tur/markdown was ported from, over 30,000 random documents.
  • Setup: node, and commonmark.js installed from the pinned lockfile: (cd tests/markdown-spec/oracle && npm ci).
  • The reference: tests/markdown-spec/oracle/render.js patches the places where commonmark.js departs from the spec, following cmark there, and says why at each. So every difference it reports is a finding.
• The GFM extensions, against cmark-gfm, GitHub's renderer, over 30,000 random documents made of table rows, tildes, URLs, email addresses and task boxes.
  • Setup: a checkout of github/cmark-gfm at ../cmark-gfm, at the commit in tests/markdown-spec/oracle/cmark-gfm.pin (--cmark-gfm DIR points elsewhere), and cmake. The script builds it, with tests/markdown-spec/oracle/gfmrender.c, in a temporary directory, which takes about 15 seconds.
  • What it leaves out: cmark-gfm's core is CommonMark 0.29. So a document is compared only when the two renderers agree on it with the extensions off, both as written and with its tildes removed. The summary line says how many documents that leaves out.

Each comparison PASS-skips on its own when its reference is missing. CI's aux part sets up both.

• Arguments go to tests/markdown-spec/diff.py:
  • --seeds 4,5 and --docs N pick the random documents (--docs per seed, for each comparison);
  • --no-corpus leaves out the files;
  • --show N prints the first N differences, with the input.

tests/check-mdrender.sh

The docs generators' Markdown renderer (ctest tur_md_render): tools/md/render.tur, built and run by tools/mdrender.py. It renders a few pages in one batch and checks the HTML, the headings the page chrome is built from, and the raw-HTML report the guides' build fails on. It takes tur as its argument, else $TUR, else ./build/tur. The first run builds the renderer, which takes a few seconds; later runs reuse the build until tur or a stdlib source changes.

tests/run-flags.sh

The global flags and the CLI's own output (ctest tur_flags_tests): inline shell cases, not fixtures, so it takes no filter. TUR=<binary> points it at another build. It ends with goldens under tests/flags-golden/ that pin tur --help, every subcommand's --help, the "external commands" listing, both completion scripts, and how the global flags are stripped. When a change to the CLI's text is intended, rewrite them and review the diff:

TUR_FLAGS_GOLDEN_UPDATE=1 timeout 720 bash tests/run-flags.sh
git diff tests/flags-golden/

ctest

timeout 720 ctest --test-dir build -R 'r7rs' -E 'conformance'

-R and -E are regexes on the target name. CI's parts (below) are exactly such patterns.

3. Caching, parallelism and binaries

4. Variants and gates

Leak checking of emitted programs is a separate harness, tests/run-leak-check.sh, for fixtures carrying requires.leak-check. run.sh checks tur itself for leaks, not the programs it compiles. See test-suite-portability-guide.md.

5. Fixture files and markers

A fixture is a directory under tests/fixtures/ (negative fixtures under tests/fixtures/errors/). These are the files the three fixture harnesses read. A file not in this table is ignored, so a misspelled marker silently removes a check.

What the fixture is

What it must do

When it runs

CLAUDE.md, "Fixture Files", has the reasoning behind several of these.

6. Reproducing a red CI job

Every check in .github/workflows/ci.yml, and the local command that runs the same thing:

Use --leg macos for a macOS job's pattern. python3 tests/ctest-parts.py with no arguments checks that the parts between them cover every registered target.

Inside a failed job's log, look for these lines:

• FAIL <name> -- <reason> is one fixture. Rerun it alone: TUR_FORCE=1 TUR_TEST_FILTER='^<name>$' timeout 720 bash tests/run.sh.
• <N> - <target> (Failed) in ctest's summary is one target. Find its script with ctest -N -V -R '^<target>$', then run the script.
• tur build timed out (>10s) is the build budget, not a hang. It is common on the Windows runners for r7rs fixtures; give the fixture an expected.timeout.
• timed out (>15s under --interpret) is the interpreter's run budget. Check the fixture's peak memory first: the interpreter keeps about 2-4 KiB per step, and fixtures run side by side.
• passes now; delete expected.xfail means the gap the marker recorded has closed. Delete the marker.

A failure on Windows or macOS that you cannot reproduce on Linux is often timing. Before calling it a product bug, check whether the failing fixture passed on that platform in an earlier run of the same shard.

7. Keeping this guide true

tests/check-running-tests-guide.sh (part of Docs lint) fails when tests/run.sh, tests/run-turi.sh or tests/run-jit.sh reads a TUR_*/TURI_* variable with a default that this guide does not mention. Adding a variable to a harness means adding a row here.