Writing a provider¶
A provider implements one or more of ods-sdk's contracts: reading a warehouse's
relations, analyzing SQL, storing state, running a build. Everything vendor-specific
lives in a provider; ODS's core and modules only see the contract
(ADR-0006).
To prove a provider keeps a contract, ods-sdk ships a conformance suite for each
one. The providers in this repository run them, and a provider written elsewhere runs
the same suites from its own tests (#99).
Contracts and their suites¶
| Contract (trait) | Suite (ods_sdk::conformance::…) |
Harness | run |
Reference fake |
|---|---|---|---|---|
ChangeProvider |
changes |
ChangeHarness |
async | FakeChangeProvider |
Credentials |
credentials |
CredentialsHarness |
async | FakeCredentials |
HealthCheck |
health_check |
HealthCheckHarness |
async | FakeHealthCheck |
ErrorCatalogue |
error_catalogue |
ErrorCatalogueHarness |
sync | FakeErrorCatalogue |
Executor |
executor |
ExecutorHarness |
async | FakeExecutor |
LockProvider |
lock |
LockHarness |
async | FakeLockProvider |
ObservedLineageSource |
observed_lineage |
ObservedLineageHarness |
sync | FakeObservedLineageSource |
RelationProbe |
probe |
ProbeHarness |
async | FakeRelationProbe |
RelationPrivileges |
privileges |
PrivilegesHarness |
async | FakeRelationPrivileges |
RelationLinker |
relation_link |
RelationLinkHarness |
sync | FakeRelationLinker |
RelationInspector |
relations |
RelationHarness |
async | FakeExecutor |
SecretProvider |
secrets |
SecretsHarness |
async | FakeSecrets |
SqlLineageAnalyzer |
sql_lineage |
SqlLineageHarness |
sync | FakeSqlLineageAnalyzer |
StateStore |
state_store |
StateStoreHarness |
async | FakeStateStore |
The fakes are in ods-provider-fake; read one next to its suite to see the smallest
implementation that passes.
Running a suite¶
A suite needs a harness: a small type in your tests that hands the suite fresh
instances of your provider, plus whatever the contract needs to be exercised without a
live platform (a fixture, a temp directory, a way to move time forward). The suite
runs every case and returns a Report:
// tests/conformance.rs in your provider crate
use std::sync::Arc;
use ods_sdk::conformance::sql_lineage::{SqlLineageHarness, run};
use ods_sdk::contracts::sql_lineage::SqlLineageAnalyzer;
struct Harness;
impl SqlLineageHarness for Harness {
fn analyzer(&self) -> Arc<dyn SqlLineageAnalyzer> {
Arc::new(my_provider::MyAnalyzer::new())
}
}
#[test]
fn conforms() {
let report = run(&Harness);
assert!(report.skipped.is_empty(), "{report:?}");
}
For an async suite, make the test #[tokio::test] and .await the run.
- A failing case panics with the contract and what it expected, so the test output says which rule was broken.
- Capabilities decide what runs. A case that needs a capability your provider
doesn't advertise is skipped and listed in
report.skippedwith the reason, so a provider is never tested for behaviour it doesn't claim. Assert on the skips you expect, so that a capability you meant to advertise and didn't shows up as a failure. - No network. Suites run on fixtures and fakes. A harness for a warehouse-backed provider runs on recorded responses or a local stand-in, never a live account.
Depending on the SDK¶
The suites are behind ods-sdk's conformance feature, so only tests pull them in.
ODS's crates are not published to crates.io yet, so depend on the repository, pinned to
a release tag:
[dependencies]
ods-sdk = { git = "https://github.com/buchochelliq-labs/open-data-suite", tag = "v0.0.1" }
[dev-dependencies]
ods-sdk = { git = "https://github.com/buchochelliq-labs/open-data-suite", tag = "v0.0.1", features = ["conformance"] }
The sql_lineage and observed_lineage suites are newer than v0.0.1: until the next
release, pin a commit on main with rev = "…" instead of tag.
Before 1.0 a contract accepts only its exact minor version, and SDK_VERSION changes
whenever a contract does; the changelog
lists each change under Breaking with what to do. Move to a new tag and rerun the
suites.
A custom ods (in-process plugins)¶
A provider plugs into ODS through a custom ods: a small binary crate of your own
that runs the released CLI with your plugins added
(ADR-0031). Three kinds of plugin go in this way:
- Health checks. Any
HealthCheck(ADR-0030).ods health checkruns it with the built-in, declared and probe checks, and the dashboard shows its recorded findings. - Warehouse plugins. A
WarehousePluginserves one warehouse kind, as the project names it (for dbt, the manifest'sadapter_type, e.g.snowflake), the way dbt picks its adapter. It never connects on its own. Every method is optional: changes: aChangeProviderover the connection dbt already has (aRelationProbe), which reads sources' data versions forods state;privileges: aPrivilegedProbe(aRelationProbethat is alsoRelationPrivileges), which reports what the probe login may do, so probe checks can run without--allow-elevated-login;links: aRelationLinkerfor "Open in warehouse", built from the warehouse's[providers.<name>]settings (WarehouseSettings, with secret references left unresolved), or the reason there are no links;observed_lineage: anObservedLineageSourcereading what the user exported from the warehouse, forods lineage compare --observed;errors: anErrorCatalogueof the warehouse's own messages, asked before dbt's when a node fails; dbt's steps come with what it recognises (error patterns);dialect: the SQL dialect column lineage parses its SQL in, by the shared parser's name for it (a name the parser doesn't know refuses the plugin);credentials: how ODS signs in to the warehouse itself, aCredentialsfor one configured provider, with secret references resolved through theSecretResolverit is given only when a token is needed. Building it does no I/O;ods doctor --connectcalls itsstatus(connectivity.warehouse_login);settings: the keys a[providers.<name>]of its kind may set undersettings, each with what it is for. With a list, any other key is reported with a warning on stderr when configuration loads, so a misspelt setting isn't silently ignored (from 0.2.0 it is a configuration error,ODS-E0102); without one, nothing is checked.
A plugin can also name the warehouses it is built on, nearest first (parents, as a
dbt adapter names the adapters it depends on). Their plugins' error patterns and
dialect then apply to it too, after its own, as dbt's adapter dispatch looks for a
macro. Nothing else is inherited. A project can say the same for a warehouse with no
plugin, with [warehouses.<kind>] extends (docs/cli.md).
What a plugin offers is detected, never declared: ODS calls each method, over a
connection that refuses every statement and with empty settings, and lists what
comes back. The released ods has one built-in warehouse plugin, Databricks: Delta
table versions, Unity Catalog's login check, Catalog Explorer links, Unity Catalog's
column lineage exports, dbt-databricks's error messages and the Databricks dialect.
- Secret providers. A SecretProvider serves one scheme of the secret references
in configuration, { secret = "<scheme>:<name>" }: an OS keychain (keychain:), a
cloud secret store (vault:, aws-sm:, …), or anything else that holds
credentials (ADR-0032). Add one with
.secret_provider(ods_cli::origin!(), Arc::new(MyStore)). env: (environment
variables, and so CI secrets) is built in and reserved. ODS resolves a
reference only where a credential is used and keeps the value in memory for that
use alone; a SecretValue can't be printed, serialised or cloned. Your provider
returns NotFound for a missing secret and Empty for an empty one, never an empty
value, and its errors name the reference, never the value: the secrets suite
checks both (give its harness an empty secret, or the Empty case is skipped).
// src/main.rs of your crate
use std::process::ExitCode;
use std::sync::Arc;
fn main() -> ExitCode {
let ods = ods_cli::Ods::new() // the released `ods`
.health_check(ods_cli::origin!(), Arc::new(my_checks::OwnerTagged))
.and_then(|ods| ods.warehouse(Arc::new(my_warehouse::Plugin)));
match ods {
Ok(ods) => ods.run(),
Err(e) => {
eprintln!("error: {e}");
ExitCode::FAILURE
}
}
}
- Registration is strict. A health check needs a valid id that no other plugin
check has. A secret scheme must be lowercase letters, digits or
-, and one no other provider serves. A warehouse already served, by a built-in or another plugin, is refused. To replace one, say so withreplacing_warehouse: the replacement serves the warehouse alone, so offer again what the built-in did if you want to keep it (examples/custom-odsreplaces the built-induckdbplugin and keeps its error patterns and dialect fromods-provider-duckdb). A dialect the parser doesn't know is refused either way. - Visible.
ods versionandods doctor(capabilities.plugins) list every plugin, with each contract it was detected to implement and its crate, and say which are built in. - Configured like the built-ins.
[health.plugins.<id>]takesseverity(includingoff),selectandexclude(docs/cli.md). An id no plugin check has is a configuration error. - Versions. Your crate and ODS must use the same
ods-sdk, so pin ODS to a release tag, as above. A plugin built against another SDK doesn't compile in.
examples/custom-ods is a complete example, built in this repository's CI. It adds a
health check (custom.owner_tagged) and a warehouse plugin for duckdb, passes both
conformance suites, and tests the binary end to end. Copy it to start.
Built-in warehouses¶
What the released ods knows about each warehouse it has a plugin for, as ods plugin
list and ods plugin show <kind> report it (detected by asking the plugin, never as it
declares). A warehouse without a plugin works too: dbt runs it, ODS reads dbt's error
patterns and maps the dialect from its kind, and the rest is not offered. A warehouse
built on another (Built on, or [warehouses.<kind>] extends) also takes that one's
error patterns and dialect (ADR-0031 §3b).
| Warehouse | Source versions | Login check | Links | Observed lineage | Error patterns | Dialect | Built on |
|---|---|---|---|---|---|---|---|
databricks |
table version from the Delta history | yes | yes | yes | databricks catalogue 1 | databricks | spark |
duckdb |
— | — | — | — | duckdb catalogue 1 | duckdb | — |
Writing a source-version provider¶
A source-version provider is a ChangeProvider, offered by a warehouse plugin's
changes. ods state asks it about every source of the project and decides, from the
answers, which models can be reused (ADR-0022).
- A version is a promise. Equal values must mean the same data. Read the version
through the
RelationProbeyou are given: one or more read-only statements per source's relation, run on dbt's own connection. - Grade it honestly. Each version carries an
Exactness, and onlyexactandsemanticversions let ODS reuse a node:
| Grade | Means | Example | Reuse |
|---|---|---|---|
exact |
identifies the exact data | a Delta table's id and version | yes |
semantic |
changes whenever the data changes in the sense that matters | the latest load batch of an append-only table | yes |
proxy |
correlated, but can move without a data change, or miss one | a last-modified time that moves on metadata changes | no |
Grade by what the warehouse documents, not by what is usually true. When unsure,
pick the lower grade: a lower grade costs a rebuild, a higher one can reuse stale
data.
- Fail to unknown (AGENTS rule 3). A source you can't read is Unknown, with the
reason (not a table, no history, no access), and its readers build. Err means you
read nothing at all, and every source is unknown. Never guess a version.
- Run the changes suite over ods-provider-fake's FakeRelationProbe. Give it
rows for your statements, and have commit change them.
examples/custom-ods/tests/conformance.rs does this for its LoadBatches.
Out-of-process plugins¶
A plugin can also be a program, in any language, that the released ods starts and
talks to over its stdin and stdout (ADR-0031
§5). A project or user declares it with [plugins.<name>]
(docs/cli.md). It offers health checks
(health_check), and for one warehouse: source versions (change_provider), links
(relation_linker), error explanations (error_catalogue), observed lineage
(observed_lineage_source), probes (relation_probe) and what the login it probes with
may do (relation_privileges), and the warehouse's SQL dialect.
Every call starts the program once, writes one JSON request to its stdin and reads one
JSON answer from its stdout, as script checks do: the envelope carries the protocol
version, and an answer in another major is refused. The types are in ods_sdk::protocol,
so a Rust plugin can use them as they are. What it writes to stderr is logged. Each call
is bounded by the plugin's timeout; it is then stopped, with every process it started.
describe, sent first:
{"protocol": {"major": 1, "minor": 0}, "request": "describe"}
The answer names the plugin and each contract it implements, with the contract version
it was built against (ods_sdk::contracts::health_check::HEALTH_CHECK.version, …), its
checks, and the warehouse kind it serves (as dbt names it, adapter_type):
{"protocol": {"major": 1, "minor": 0}, "name": "acme", "version": "1.2.0",
"provides": [{"contract": "health_check", "version": {"major": 0, "minor": 2}},
{"contract": "change_provider", "version": {"major": 0, "minor": 1}}],
"checks": [{"id": "acme.pii_tagged", "about": "PII columns are tagged",
"default_severity": "warn"}],
"warehouse": "snowflake", "versions_read": "Snowflake table versions",
"capabilities": ["relation_versions"]}
A contract version the host can't accept (before 1.0, any other minor) refuses the
plugin, naming both. Checks without health_check, a check listed twice, or
change_provider without a warehouse refuse it too. So does change_provider
whose capabilities name neither relation_versions (a table's own version) nor
source_freshness (its last load): ods state chooses among readings by the kind of
version they are, so one that doesn't say would be read and never used. A contract the host doesn't know
is listed and otherwise ignored, so a plugin can offer more than an older ods uses.
A warehouse another plugin already serves (e.g. a built-in) is a configuration error.
check, for each of its checks, is the script protocol's
request, with the check's id and no params: answer one finding per node in scope,
and none for any other.
versions asks for sources' current data versions, in one batch, with the
settings of each [providers.<name>] of the plugin's warehouse kind, by name (secret references stay
references: resolve them yourself, from your own environment):
{"protocol": {"major": 1, "minor": 0}, "request": "versions",
"sources": [{"id": "source.shop.raw.orders", "name": "raw.orders"}],
"settings": {"warehouse": {"account": "acme", "password": {"secret": "env:SF_PASSWORD"}}}}
Answer each source with a version (graded as in
Writing a source-version provider) or unknown
and why:
{"protocol": {"major": 1, "minor": 0},
"sources": [{"id": "source.shop.raw.orders",
"version": {"value": "01HZX…", "exactness": "exact", "source": "acme"}},
{"id": "source.shop.raw.events", "unknown": "a view has no version"}]}
A source left out is unknown. An answer about a source it wasn't asked about, or about one twice, makes every source in the call unknown. An out-of-process plugin can't use dbt's connection: it connects with its own credentials, from its own environment.
The warehouse requests (ADR-0031 §3d) each
carry the same settings, and are answered in one call:
| Request | Asked | Answer |
|---|---|---|
link |
relations: every relation of the project, as dbt renders it |
links: {"relation", "link": {"url", "label"}} or {"relation", "no_link": {"reason", …}} each |
classify |
errors: failed nodes' ErrorSummarys (values already removed) |
classifications: one Classification each, in order |
observed_lineage |
export: an absolute path the user named |
lineage: the ObservedLineage it read, or exit non-zero |
probe |
probe (a ProbeRequest) and targets (ProbeTargets, each with the relation the build made) |
targets: {"id", "answer": {"rows": […]} | {"skipped": why} | {"unknown": why}} each |
privileges |
targets |
login, and targets: {"id", "access": "read_only" | {"elevated": […]} | {"unknown": why}} each |
describe names the dialect (a name the shared parser knows) and, for
error_catalogue, its catalogue ({"name", "version", "engine"}).
relation_privileges needs relation_probe: only the connection that probes can say
what its login may do. A link must be an https:// URL with a label and no user part
or fragment; anything else, a relation it doesn't answer about, or a failed call is no
link. An error it doesn't recognise, or a failed call, is left to dbt's catalogue. A
probe or privilege answer about a target it wasn't asked about, or about one twice,
makes the whole call unknown.
ODS asks for links and explanations from the commands that run plugins, and keeps the answers for the dashboard and MCP, which never start one. So a plugin is asked about everything at once, and only when something changed: every relation when a command reads the project, each error once.
Testing it: ods plugin test <name> runs the SDK's suites against the plugin as ODS
calls it: health_check for each check, over fixture nodes, and changes over the
sources you give with --source (two it can read) and --unreadable (one it can't).
Run it before publishing, and in your plugin's CI:
ods plugin test acme --allow-scripts \
--source source.shop.raw.orders --source source.shop.raw.customers \
--unreadable source.shop.raw.events_view
In CI¶
A job that runs the suites on every push:
name: conformance
on: [push, pull_request]
jobs:
conformance:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: dtolnay/rust-toolchain@stable
- run: cargo test --test conformance