Testing¶
Run everything before opening a pull request:
cargo fmt --all --check
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace
ruff check .
mypy
pytest # needs the package installed (maturin)
(cd vscode-extension && npm test)
python3 .github/rab/scripts/release.py check
python3 scripts/gen_cli_reference.py --check
zensical build --strict # the documentation
CI (.github/workflows/ci.yml) runs the same checks on every pull request, and the release
workflow reruns them on the tagged commit.
What the suites cover¶
| Suite | Where | What |
|---|---|---|
| Rust unit tests | crates/verlato/src/** (#[cfg(test)]) |
classification, grouping, rendering, state migrations, providers (with fake servers), the service and MCP protocols, pyjson |
| Extension points | crates/verlato/tests/extensions.rs |
an out-of-crate extension registering one of every point |
| Python package | tests/test_python_package.py |
the bindings: run, call, ServiceError, the packaged parser bundle, version agreement with Cargo |
| Parity | tests/test_rust_parity.py + tests/golden/parity/ |
the native runtime against recorded answers of the retired Python implementation |
| Extension | vscode-extension/test/*.test.js |
webview markup, CLI resolution, model environment, semantic evidence rendering, cancellation |
The golden parity suite¶
Verlato was first written in Python. Before the Python implementation was retired, its
answers to every scenario in tests/test_rust_parity.py (command output, files written,
requests sent to fake model servers, MCP sessions) were recorded in tests/golden/parity/.
The suite now replays each scenario against the native runtime and requires the same answers,
byte for byte, apart from documented differences (engine version strings, and the repository
README that the MCP guidance resources serve). See tests/golden/README.md for the format.
New behaviour cannot be recorded there any more: test it with expected values in Rust unit
tests or tests/test_python_package.py.
Determinism¶
Tests make no network calls: model providers are tested against local fake servers and the
mock provider; git fixtures use fixed authors and dates, so commit IDs are stable. The
IntentumDiff parser bundle is needed for engine tests (scripts/fetch_intentumdiff_parsers.sh).