Skip to content

Writing a provider

A provider implements one or more of ods-sdk's contracts: reading a warehouse's relations, analyzing SQL, storing state, running a build. Everything vendor-specific lives in a provider; ODS's core and modules only see the contract (ADR-0006).

To prove a provider keeps a contract, ods-sdk ships a conformance suite for each one. The providers in this repository run them, and a provider written elsewhere runs the same suites from its own tests (#99).

Contracts and their suites

Contract (trait) Suite (ods_sdk::conformance::…) Harness run Reference fake
ChangeProvider changes ChangeHarness async FakeChangeProvider
Credentials credentials CredentialsHarness async FakeCredentials
HealthCheck health_check HealthCheckHarness async FakeHealthCheck
ErrorCatalogue error_catalogue ErrorCatalogueHarness sync FakeErrorCatalogue
Executor executor ExecutorHarness async FakeExecutor
LockProvider lock LockHarness async FakeLockProvider
ObservedLineageSource observed_lineage ObservedLineageHarness sync FakeObservedLineageSource
RelationProbe probe ProbeHarness async FakeRelationProbe
RelationPrivileges privileges PrivilegesHarness async FakeRelationPrivileges
RelationLinker relation_link RelationLinkHarness sync FakeRelationLinker
RelationInspector relations RelationHarness async FakeExecutor
SecretProvider secrets SecretsHarness async FakeSecrets
SqlLineageAnalyzer sql_lineage SqlLineageHarness sync FakeSqlLineageAnalyzer
StateStore state_store StateStoreHarness async FakeStateStore

The fakes are in ods-provider-fake; read one next to its suite to see the smallest implementation that passes.

Running a suite

A suite needs a harness: a small type in your tests that hands the suite fresh instances of your provider, plus whatever the contract needs to be exercised without a live platform (a fixture, a temp directory, a way to move time forward). The suite runs every case and returns a Report:

// tests/conformance.rs in your provider crate
use std::sync::Arc;

use ods_sdk::conformance::sql_lineage::{SqlLineageHarness, run};
use ods_sdk::contracts::sql_lineage::SqlLineageAnalyzer;

struct Harness;

impl SqlLineageHarness for Harness {
    fn analyzer(&self) -> Arc<dyn SqlLineageAnalyzer> {
        Arc::new(my_provider::MyAnalyzer::new())
    }
}

#[test]
fn conforms() {
    let report = run(&Harness);
    assert!(report.skipped.is_empty(), "{report:?}");
}

For an async suite, make the test #[tokio::test] and .await the run.

  • A failing case panics with the contract and what it expected, so the test output says which rule was broken.
  • Capabilities decide what runs. A case that needs a capability your provider doesn't advertise is skipped and listed in report.skipped with the reason, so a provider is never tested for behaviour it doesn't claim. Assert on the skips you expect, so that a capability you meant to advertise and didn't shows up as a failure.
  • No network. Suites run on fixtures and fakes. A harness for a warehouse-backed provider runs on recorded responses or a local stand-in, never a live account.

Depending on the SDK

The suites are behind ods-sdk's conformance feature, so only tests pull them in. ODS's crates are not published to crates.io yet, so depend on the repository, pinned to a release tag:

[dependencies]
ods-sdk = { git = "https://github.com/buchochelliq-labs/open-data-suite", tag = "v0.0.1" }

[dev-dependencies]
ods-sdk = { git = "https://github.com/buchochelliq-labs/open-data-suite", tag = "v0.0.1", features = ["conformance"] }

The sql_lineage and observed_lineage suites are newer than v0.0.1: until the next release, pin a commit on main with rev = "…" instead of tag.

Before 1.0 a contract accepts only its exact minor version, and SDK_VERSION changes whenever a contract does; the changelog lists each change under Breaking with what to do. Move to a new tag and rerun the suites.

A custom ods (in-process plugins)

A provider plugs into ODS through a custom ods: a small binary crate of your own that runs the released CLI with your plugins added (ADR-0031). Three kinds of plugin go in this way:

  • Health checks. Any HealthCheck (ADR-0030). ods health check runs it with the built-in, declared and probe checks, and the dashboard shows its recorded findings.
  • Warehouse plugins. A WarehousePlugin serves one warehouse kind, as the project names it (for dbt, the manifest's adapter_type, e.g. snowflake), the way dbt picks its adapter. It never connects on its own. Every method is optional:
  • changes: a ChangeProvider over the connection dbt already has (a RelationProbe), which reads sources' data versions for ods state;
  • privileges: a PrivilegedProbe (a RelationProbe that is also RelationPrivileges), which reports what the probe login may do, so probe checks can run without --allow-elevated-login;
  • links: a RelationLinker for "Open in warehouse", built from the warehouse's [providers.<name>] settings (WarehouseSettings, with secret references left unresolved), or the reason there are no links;
  • observed_lineage: an ObservedLineageSource reading what the user exported from the warehouse, for ods lineage compare --observed;
  • errors: an ErrorCatalogue of the warehouse's own messages, asked before dbt's when a node fails; dbt's steps come with what it recognises (error patterns);
  • dialect: the SQL dialect column lineage parses its SQL in, by the shared parser's name for it (a name the parser doesn't know refuses the plugin);
  • credentials: how ODS signs in to the warehouse itself, a Credentials for one configured provider, with secret references resolved through the SecretResolver it is given only when a token is needed. Building it does no I/O; ods doctor --connect calls its status (connectivity.warehouse_login);
  • settings: the keys a [providers.<name>] of its kind may set under settings, each with what it is for. With a list, any other key is reported with a warning on stderr when configuration loads, so a misspelt setting isn't silently ignored (from 0.2.0 it is a configuration error, ODS-E0102); without one, nothing is checked.

A plugin can also name the warehouses it is built on, nearest first (parents, as a dbt adapter names the adapters it depends on). Their plugins' error patterns and dialect then apply to it too, after its own, as dbt's adapter dispatch looks for a macro. Nothing else is inherited. A project can say the same for a warehouse with no plugin, with [warehouses.<kind>] extends (docs/cli.md).

What a plugin offers is detected, never declared: ODS calls each method, over a connection that refuses every statement and with empty settings, and lists what comes back. The released ods has one built-in warehouse plugin, Databricks: Delta table versions, Unity Catalog's login check, Catalog Explorer links, Unity Catalog's column lineage exports, dbt-databricks's error messages and the Databricks dialect. - Secret providers. A SecretProvider serves one scheme of the secret references in configuration, { secret = "<scheme>:<name>" }: an OS keychain (keychain:), a cloud secret store (vault:, aws-sm:, …), or anything else that holds credentials (ADR-0032). Add one with .secret_provider(ods_cli::origin!(), Arc::new(MyStore)). env: (environment variables, and so CI secrets) is built in and reserved. ODS resolves a reference only where a credential is used and keeps the value in memory for that use alone; a SecretValue can't be printed, serialised or cloned. Your provider returns NotFound for a missing secret and Empty for an empty one, never an empty value, and its errors name the reference, never the value: the secrets suite checks both (give its harness an empty secret, or the Empty case is skipped).

// src/main.rs of your crate
use std::process::ExitCode;
use std::sync::Arc;

fn main() -> ExitCode {
    let ods = ods_cli::Ods::new()                         // the released `ods`
        .health_check(ods_cli::origin!(), Arc::new(my_checks::OwnerTagged))
        .and_then(|ods| ods.warehouse(Arc::new(my_warehouse::Plugin)));
    match ods {
        Ok(ods) => ods.run(),
        Err(e) => {
            eprintln!("error: {e}");
            ExitCode::FAILURE
        }
    }
}
  • Registration is strict. A health check needs a valid id that no other plugin check has. A secret scheme must be lowercase letters, digits or -, and one no other provider serves. A warehouse already served, by a built-in or another plugin, is refused. To replace one, say so with replacing_warehouse: the replacement serves the warehouse alone, so offer again what the built-in did if you want to keep it (examples/custom-ods replaces the built-in duckdb plugin and keeps its error patterns and dialect from ods-provider-duckdb). A dialect the parser doesn't know is refused either way.
  • Visible. ods version and ods doctor (capabilities.plugins) list every plugin, with each contract it was detected to implement and its crate, and say which are built in.
  • Configured like the built-ins. [health.plugins.<id>] takes severity (including off), select and exclude (docs/cli.md). An id no plugin check has is a configuration error.
  • Versions. Your crate and ODS must use the same ods-sdk, so pin ODS to a release tag, as above. A plugin built against another SDK doesn't compile in.

examples/custom-ods is a complete example, built in this repository's CI. It adds a health check (custom.owner_tagged) and a warehouse plugin for duckdb, passes both conformance suites, and tests the binary end to end. Copy it to start.

Built-in warehouses

What the released ods knows about each warehouse it has a plugin for, as ods plugin list and ods plugin show <kind> report it (detected by asking the plugin, never as it declares). A warehouse without a plugin works too: dbt runs it, ODS reads dbt's error patterns and maps the dialect from its kind, and the rest is not offered. A warehouse built on another (Built on, or [warehouses.<kind>] extends) also takes that one's error patterns and dialect (ADR-0031 §3b).

Warehouse Source versions Login check Links Observed lineage Error patterns Dialect Built on
databricks table version from the Delta history yes yes yes databricks catalogue 1 databricks spark
duckdb — — — — duckdb catalogue 1 duckdb —

Writing a source-version provider

A source-version provider is a ChangeProvider, offered by a warehouse plugin's changes. ods state asks it about every source of the project and decides, from the answers, which models can be reused (ADR-0022).

  • A version is a promise. Equal values must mean the same data. Read the version through the RelationProbe you are given: one or more read-only statements per source's relation, run on dbt's own connection.
  • Grade it honestly. Each version carries an Exactness, and only exact and semantic versions let ODS reuse a node:
Grade Means Example Reuse
exact identifies the exact data a Delta table's id and version yes
semantic changes whenever the data changes in the sense that matters the latest load batch of an append-only table yes
proxy correlated, but can move without a data change, or miss one a last-modified time that moves on metadata changes no

Grade by what the warehouse documents, not by what is usually true. When unsure, pick the lower grade: a lower grade costs a rebuild, a higher one can reuse stale data. - Fail to unknown (AGENTS rule 3). A source you can't read is Unknown, with the reason (not a table, no history, no access), and its readers build. Err means you read nothing at all, and every source is unknown. Never guess a version. - Run the changes suite over ods-provider-fake's FakeRelationProbe. Give it rows for your statements, and have commit change them. examples/custom-ods/tests/conformance.rs does this for its LoadBatches.

Out-of-process plugins

A plugin can also be a program, in any language, that the released ods starts and talks to over its stdin and stdout (ADR-0031 §5). A project or user declares it with [plugins.<name>] (docs/cli.md). It offers health checks (health_check), and for one warehouse: source versions (change_provider), links (relation_linker), error explanations (error_catalogue), observed lineage (observed_lineage_source), probes (relation_probe) and what the login it probes with may do (relation_privileges), and the warehouse's SQL dialect.

Every call starts the program once, writes one JSON request to its stdin and reads one JSON answer from its stdout, as script checks do: the envelope carries the protocol version, and an answer in another major is refused. The types are in ods_sdk::protocol, so a Rust plugin can use them as they are. What it writes to stderr is logged. Each call is bounded by the plugin's timeout; it is then stopped, with every process it started.

describe, sent first:

{"protocol": {"major": 1, "minor": 0}, "request": "describe"}

The answer names the plugin and each contract it implements, with the contract version it was built against (ods_sdk::contracts::health_check::HEALTH_CHECK.version, …), its checks, and the warehouse kind it serves (as dbt names it, adapter_type):

{"protocol": {"major": 1, "minor": 0}, "name": "acme", "version": "1.2.0",
 "provides": [{"contract": "health_check", "version": {"major": 0, "minor": 2}},
              {"contract": "change_provider", "version": {"major": 0, "minor": 1}}],
 "checks": [{"id": "acme.pii_tagged", "about": "PII columns are tagged",
             "default_severity": "warn"}],
 "warehouse": "snowflake", "versions_read": "Snowflake table versions",
 "capabilities": ["relation_versions"]}

A contract version the host can't accept (before 1.0, any other minor) refuses the plugin, naming both. Checks without health_check, a check listed twice, or change_provider without a warehouse refuse it too. So does change_provider whose capabilities name neither relation_versions (a table's own version) nor source_freshness (its last load): ods state chooses among readings by the kind of version they are, so one that doesn't say would be read and never used. A contract the host doesn't know is listed and otherwise ignored, so a plugin can offer more than an older ods uses. A warehouse another plugin already serves (e.g. a built-in) is a configuration error.

check, for each of its checks, is the script protocol's request, with the check's id and no params: answer one finding per node in scope, and none for any other.

versions asks for sources' current data versions, in one batch, with the settings of each [providers.<name>] of the plugin's warehouse kind, by name (secret references stay references: resolve them yourself, from your own environment):

{"protocol": {"major": 1, "minor": 0}, "request": "versions",
 "sources": [{"id": "source.shop.raw.orders", "name": "raw.orders"}],
 "settings": {"warehouse": {"account": "acme", "password": {"secret": "env:SF_PASSWORD"}}}}

Answer each source with a version (graded as in Writing a source-version provider) or unknown and why:

{"protocol": {"major": 1, "minor": 0},
 "sources": [{"id": "source.shop.raw.orders",
              "version": {"value": "01HZX…", "exactness": "exact", "source": "acme"}},
             {"id": "source.shop.raw.events", "unknown": "a view has no version"}]}

A source left out is unknown. An answer about a source it wasn't asked about, or about one twice, makes every source in the call unknown. An out-of-process plugin can't use dbt's connection: it connects with its own credentials, from its own environment.

The warehouse requests (ADR-0031 §3d) each carry the same settings, and are answered in one call:

Request Asked Answer
link relations: every relation of the project, as dbt renders it links: {"relation", "link": {"url", "label"}} or {"relation", "no_link": {"reason", …}} each
classify errors: failed nodes' ErrorSummarys (values already removed) classifications: one Classification each, in order
observed_lineage export: an absolute path the user named lineage: the ObservedLineage it read, or exit non-zero
probe probe (a ProbeRequest) and targets (ProbeTargets, each with the relation the build made) targets: {"id", "answer": {"rows": […]} | {"skipped": why} | {"unknown": why}} each
privileges targets login, and targets: {"id", "access": "read_only" | {"elevated": […]} | {"unknown": why}} each

describe names the dialect (a name the shared parser knows) and, for error_catalogue, its catalogue ({"name", "version", "engine"}). relation_privileges needs relation_probe: only the connection that probes can say what its login may do. A link must be an https:// URL with a label and no user part or fragment; anything else, a relation it doesn't answer about, or a failed call is no link. An error it doesn't recognise, or a failed call, is left to dbt's catalogue. A probe or privilege answer about a target it wasn't asked about, or about one twice, makes the whole call unknown.

ODS asks for links and explanations from the commands that run plugins, and keeps the answers for the dashboard and MCP, which never start one. So a plugin is asked about everything at once, and only when something changed: every relation when a command reads the project, each error once.

Testing it: ods plugin test <name> runs the SDK's suites against the plugin as ODS calls it: health_check for each check, over fixture nodes, and changes over the sources you give with --source (two it can read) and --unreadable (one it can't). Run it before publishing, and in your plugin's CI:

ods plugin test acme --allow-scripts \
  --source source.shop.raw.orders --source source.shop.raw.customers \
  --unreadable source.shop.raw.events_view

In CI

A job that runs the suites on every push:

name: conformance
on: [push, pull_request]
jobs:
  conformance:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: dtolnay/rust-toolchain@stable
      - run: cargo test --test conformance