# ADR 0011: Provider writes fail closed, and say when they may have happened Documentation version: 0.3.0. **Status:** accepted · **Date:** 2026-10-25 · **Changes:** ADR 0008 (how a provider adapter publishes a value with its policy) and ADR 0009 (what the router records about a call at a connection) ## Context The security review of 2026-10-25 found two problems at the boundary between a vault and the providers its connections reach. **SR02.** A new version with a restrictive policy (disabled, not yet valid, expired) is published before its policy: * **HashiCorp Vault** keeps the policy per version in the secret's `custom_metadata`, and a version without an entry there is enabled with no time window. The adapter writes the data, then the metadata. If the second request fails, or while it is in flight, the new version is readable. The review reproduced it with a loopback fake Vault. * **AWS Secrets Manager** keeps the policy in the secret's tags, which apply to every version. For an existing secret the adapter sends `PutSecretValue`, then the tags. A failed tag write leaves the new current version under the old policy. **SR03**, its second half. B06 made the vault check itself before every call at a connection. A call that was sent is still recorded only afterwards, and when recording fails (the vault was blocked in between, the disk is full), or the provider's answer is lost, the caller gets an error that reads as if nothing happened, although the provider may have acted. A local SQLite transaction cannot undo a remote change. ## Decision ### 1. Every outcome of a call at a connection is one of three | Outcome | When | Recorded as | Returned | |---|---|---|---| | refused | the vault's gate refused it (B06), or the provider answered with a definite refusal (`invalid_argument`, `not_found`, `deleted_conflict`, `permission_denied`, `unauthenticated`, `unsupported`, …) | `failure` or `denied`, as now | that error | | done | the provider answered with success and the vault recorded it | `success`, as now | the value | | **unknown** | the provider may have acted: its answer was lost or it failed in a way that does not prove nothing happened (`provider_unavailable` on a change), an adapter knows it acted in part, or the provider acted but the vault could not record it | **`unknown`** when the vault can still write; otherwise only the attempt below stays | **`outcome_unknown`**, a new error code | Reads are not changes: a read whose answer is lost is simply an error, and a value read but not recorded is not returned. ### 2. A change at a connection is recorded before it is sent Before a change (set, update, delete, recover, purge) is sent, once the vault's gate has passed (B06) and the connection's client is open, the router records the **attempt**: an event of the change's operation with the new outcome **`pending`** and the detail `at connection : sent`, committed before the provider is called. The event that records the result names it: `at connection [: ] (attempt )`, with `success`, `failure`, `denied` or `unknown`. An attempt is **unresolved** while nothing but an `unknown` completes it (the answer was lost, the process stopped, or the vault could not record the answer), until a later change of the same secret through the same connection succeeds (`forget_missing` included): that change was made after the secret was looked at again. `SqliteVault::unresolved_attempts` lists them, and `sourceorrery audit` names them with what to do. Nothing is retried automatically. The procedure for an unresolved attempt or an `outcome_unknown` (`docs/providers/index.md`): | Change | Safe to repeat? | Check first | |---|---|---| | `secret set` | yes: at worst a second version | `secret versions NAME` | | `secret update` | yes: it sets the same properties | `secret get NAME` (metadata) | | `secret delete` | check: a second delete is refused | `secret list --deleted` | | `secret recover` | check: a second recover is refused | `secret get NAME` | | `secret purge` | check: the catalog may still list it | `Routed::missing` / `forget_missing` | A pending attempt stays in the log for good, like every event: the log records that the vault did not learn the result, never a result it does not know. ### 3. Publishing a value never makes a restricted version readable **The invariant.** At every moment, after any failure (a refused request, a lost answer, a process that stops between two requests), and with any number of SourceOrrery writers on HashiCorp Vault, the policy a SourceOrrery reader applies to a version is at least as restrictive as the one its writer asked for: disabled if it asked for disabled, a `not_before` no earlier and an `expires_on` no later. Nothing relies on a cleanup after a failure; whatever a failure leaves behind only restricts. "Reader" means every SourceOrrery read, metadata-only ones included (properties, version lists, listings) as well as value reads with or without an explicit version. The policy binds what SourceOrrery returns. A client with its own Vault token or AWS credentials reads any version regardless; against those the control is the provider's own: a Vault ACL policy on `/data/*`, an IAM policy (or tag-based ABAC) on `secretsmanager:GetSecretValue`. **HashiCorp Vault** keeps the policy per version in `custom_metadata`. * *Representation.* A version's **committed** properties are the document `sourceorrery.v`, as before (one an earlier release wrote is a committed one). An **intent** `sourceorrery.v~` (16 random hex digits) is what one writer asked for version `N` before `N` existed. A reader applies to version `N` its committed document, or the defaults without one, **restricted by every intent for `N`**: disabled if any is, the latest `not_before`, the earliest `expires_on`; the content type, the tags and `updated_on` come from the committed document, or else from the first intent in key order. A version with neither, written by another client, is enabled with no window, as before. Releases before this one apply committed documents only. * *Writes.* Every change to `custom_metadata` is a JSON merge patch of the keys it concerns (`PATCH /metadata/`, which needs the `patch` capability), so two writers never drop each other's keys. A new secret's metadata is first created by a `POST` that sets only `max_versions: 0` and leaves `custom_metadata` alone. (Verified against Vault 2.2.1: a merge patch with `null` removes a key, a patch of missing metadata is 404, a `POST` without `custom_metadata` keeps it, and a data write whose `cas` is not the current version is refused with 400.) * *`set_secret`.* Read the metadata (current version `N`). When the properties asked for are not the defaults, patch an intent for `N+1`. Write the data with `options.cas = N`. Then one patch commits: the committed document for `N+1` (none at the defaults), this writer's intent removed, and every intent for `N+1` that the first read found removed (left by an attempt that stopped; any writer still in flight for `N+1` has lost its check-and-set). With default properties and nothing to remove, the write is the only change, as before. * *Failures of `set_secret`.* Refused before the data write, or the data write refused (another writer took `N+1`: Vault's check-and-set answers 400): nothing was written; the intent is removed if one was patched, and the refusal is returned (`invalid_argument` for the check-and-set, naming the other writer). The data write's answer lost: the intent stays, which binds `N+1` if the write landed, and the result is `outcome_unknown`. The commit failed: the version exists, held by its intent at least as restrictive as asked, and the result is `outcome_unknown`. A leftover intent restricts the version it names once someone writes it, until a later `set_secret` of that version commits or `update_secret_properties` of it decides. * *`update_secret_properties`* is one patch: the version's committed document, and its intents removed; the owner's explicit decision supersedes them. * *Delete* patches the deletion marker in first, then asks KV's delete; *recover* asks KV's undelete first, then patches the marker away. A failure of the second step leaves the marker, so SourceOrrery still treats the secret as deleted and it can be recovered or purged; the result is `outcome_unknown`. * *Limits.* Vault's `custom_metadata` limits (64 keys, 512 bytes a value) count an intent too; an intent that does not fit is refused before any data is written. **AWS Secrets Manager** keeps the policy per secret, in tags that apply to every version (`Capability::PerVersionProperties` is not claimed). * *Policy tags* are `sourceorrery:disabled`, `sourceorrery:not_before` and `sourceorrery:expires_on`; the content type, `updated_on` and user tags are the others. `TagResource` adds or overwrites keys; `UntagResource` removes them. * *`set_secret` of an existing secret.* Describe it (the old policy and tags) and validate the request (B09). When the **meet** of the old and new policy (disabled if either is, the later `not_before`, the earlier `expires_on`) is stricter than the old, tag the secret with the meet's policy tags: only additions and overwrites. Then `PutSecretValue`. Then the target, in an order that never loosens below it: remove stale non-policy tags, tag every wanted tag, remove stale policy tags last. The new version is never current under a policy looser than the target, and the old one is never under a policy looser than the meet until the new one is current; afterwards the policy of the secret is the new one, for every version, as the contract for per-secret properties says. * *`update_secret_properties`* runs the same three target steps. (They replace an order that removed stale tags first and could loosen the policy for a moment: from disabled to a future `not_before`, the secret was enabled with no window in between.) * *Failures.* The narrowing refused: nothing changed, the refusal is returned. Its answer lost: no value was sent, the policy may be narrowed, `outcome_unknown`. `PutSecretValue` refused: the narrowing is undone (old values tagged back, keys the meet added removed) and the refusal returned, or `outcome_unknown` when the undo fails. `PutSecretValue`'s answer lost, or any target step failing: `outcome_unknown`, with the secret held at least as strict as asked. A new secret gets its value and tags in one `CreateSecret`. * *Concurrent writers.* Secrets Manager has no conditional write for tags or `PutSecretValue`. Two SourceOrrery writers changing one secret at the same moment can interleave their tag steps, and a version can be under the other writer's policy until the last step lands. The invariant holds on AWS for one writer at a time per secret; a policy that must hold against concurrent writers or other clients belongs in IAM. **Azure Key Vault** sets a version's value, attributes and tags in one request: unchanged, and the invariant holds as it is. Input a provider would refuse after publication is refused before the first request: user tags, content types and the provider's limits (B09 for AWS, the existing checks for Azure, Vault's own limits at the intent). ### 4. Error code `outcome_unknown` `SecretError::OutcomeUnknown { at, reason }`, code `outcome_unknown`, appended (codes are append-only). `at` names the connection, or the provider when an adapter reports it. Not transient: a blind retry is the wrong reaction. `SecretError::is_refusal` is the list of codes that show a provider made no change, and `SecretError::after_change_sent` turns any other failure of a sent change into `outcome_unknown`; an adapter that sends more than one request for a change returns a refusal only while it has changed nothing. The Rust client's `ErrorKind` names it, as it names `rotation`. No SDK and no C ABI function changes a secret (ADR 0005), so none can meet it: the bindings type the codes a read returns, and report any other, this one included, as their base error with the code kept. `CONTRACT_VERSION` stays `"2"`: no existing scenario changes its outcome. ## Consequences * `AuditOutcome` gains `pending` and `unknown` (appended). They are part of the audit row as any outcome is; schema 5 is unreleased, so no format change follows. * A change at a connection writes two audit events instead of one. * A Vault write with properties that are not the defaults makes four requests instead of three (read, intent, data, commit), and needs the `patch` capability on `/metadata/*`; a token without it is refused with `permission_denied` before any data is written. * An AWS write with a policy change makes up to five requests; the old current version is restricted for the moment between them when the new policy is stricter. * On AWS, the invariant needs one writer at a time per secret (above). ## Alternatives considered * **Swap the two requests** (Vault metadata first, AWS tags first). Correct for a stricter new policy, wrong for a looser one (AWS: the old version would be loosened early), and silent about lost answers. * **Vault: write the committed document for `N+1` before the data, replacing the whole `custom_metadata` map.** Correct for one writer, but a second writer's read-modify-write of the map (a property update of another version, a deletion marker, its own pre-write) can drop the entry, and the version is then enabled when its data lands. Per-writer intents patched key by key cannot be dropped that way. * **Vault: treat a version without properties as disabled.** Fails closed without intents, but every version another client writes would become unreadable through SourceOrrery. * **Keep the policy inside the Vault data** (one atomic write per version). Metadata listings would have to read values, and property updates still need `custom_metadata`. * **Clean up after a failed second request.** The version is readable until the cleanup lands, and the cleanup can fail too. * **A distributed transaction or a write-ahead log replayed on open.** The providers offer no transaction with SQLite, and replaying a change whose outcome is unknown could repeat it.