# Trusty payload format -- v1 + v2 Canonical byte-level specification of what goes into `raw_payload`, what `row_hash` commits to, and how v2 signatures are produced and verified. This document is consensus-critical. v1 byte-shape is locked by the invariant tests in `trusty-core` (see tag `s5_invariant_v1_byte_shape_preserved` and the `v1_*` tests in `payload.rs` / `chain.rs`). v2 is purely additive and must preserve v1 bytes exactly when `payload_version=1`. The original Python port-source is archival only and no longer gates changes here. Source of truth in-tree: - `crates/trusty-core/src/payload.rs` -- canonical serializer, constants. - `crates/trusty-core/src/chain.rs` -- `compute_record`, `compute_record_signed`, `verify_links`, `verify_row_signatures`. - `crates/trusty-core/src/hash.rs` -- `row_hash`, `anchor_hash`. - `crates/trusty-core/src/crypto.rs` -- `Signer`, `Verifier`, Ed25519 impls. --- ## 1. Canonical form Every `raw_payload` is a JSON object. Its canonical byte form is: 1. **Keys sorted alphabetically**, recursively (every nested object gets the same treatment). Implemented via `BTreeMap` at the top level and a recursive `canonicalize` on all nested `serde_json::Value::Object` nodes. 2. **No whitespace** -- `serde_json::to_string` (not `to_string_pretty`). 3. **UTF-8**. No BOM. 4. **Array order preserved**. Only object keys are sorted; arrays keep caller order (this matters for `signatures`, which is an array -- but each entry inside is itself an object and gets its keys sorted). ### 1.1 Field list (alphabetical -- the on-wire order) | Field | Type | Notes | |---------------------|------------|-----------------------------------------------------------------------| | `chain_id` | string | `chain__<6-digit-generation>` (e.g. `chain_acme_000001`). | | `data` | any JSON | Caller-provided. Canonicalized recursively. Capped at 65,536 bytes after canonicalization. | | `detected_at` | string | ISO-8601 UTC. Time trusty observed the event. | | `entity_id` | string | Opaque caller identifier. | | `extraction_method` | string | Caller tag for the provenance source (e.g. `trusty`, `scraper-v3`). | | `hash_alg` | string | Always `"siphash24+sha256"` today. Verifiers read this before recomputing `row_hash`. | | `payload_version` | number | `1` (v1) or `2` (v2). | | `prev_hash` | string | Previous row's `row_hash` (16 hex chars), or the literal `"genesis"` on a segment's first row. | | `received_at` | string | ISO-8601 UTC with sub-second precision. Distinct from `detected_at`; never collapse. | | `sequence` | number | Per-chain monotonically increasing `i64`. Genesis = 0. | | `signatures` | array | v1: always `[]`. v2: one object per signer (see Section 3). | | `source_url` | string | Where the data came from. | ### 1.2 `data` size cap `MAX_DATA_BYTES = 65_536` enforced after canonicalization. Exceeding it raises `TrustyError::DataTooLarge`. Put blobs in object storage and reference them by `{uri, sha256}` inside `data` -- do not inline them. --- ## 2. Dual hash Each row carries two hashes. Both are computed in `trusty-core/src/hash.rs` and are byte-identical to the Python reference. ### 2.1 `row_hash` -- SipHash-2-4 - Keyed with the 16-byte `SecretKey` (split into `k0` / `k1` as two little-endian `u64`s, `bytes[0..8]` and `bytes[8..16]`). - Input: the exact canonical payload bytes (`raw_payload` above). - Output: 64-bit digest rendered as **lowercase hex**, **16 chars**, wire representation derived from `format!("{:016x}", digest)`. This is a big-endian-style hex rendering of the integer value. **BE hex is canonical; any legacy LE rendering (e.g. earlier threadkeeper code) is non-canonical and must be adapter-converted when crossing the boundary.** See tag `s2_decision_be_encoding_canonical`. ### 2.2 `anchor_hash` -- SHA-256 Input byte layout: ``` row_hash_bytes || 0x00 || entity_id_bytes || 0x00 || chain_id_bytes ``` All three text fields are UTF-8. Output is lowercase hex, **64 chars**. Null bytes are domain separators between fields -- do not drop them. The dual-hash shape (keyed SipHash + keyless SHA-256 with null-byte domain separation) is locked in `trusty-core/src/hash.rs` and covered by the `anchor_hash_*` tests there. It is the only hash layout trusty recognises. ### 2.3 Why two hashes SipHash-2-4 is a keyed fingerprint: cheap, fast, tamper-evident against adversaries who do not hold the key. It is the chain-integrity primitive. SHA-256 is the cryptographic anchor: public, collision-resistant, suitable for external receipts and cross-system reference. The anchor is what you publish or notarize; the row hash is what you use to link the chain. Dual hash is non-negotiable -- do not collapse to one. See `CLAUDE.md#design-rules`. --- ## 3. v2 signatures v2 adds authenticated signatures over the payload. v1 is unchanged. ### 3.1 Shape of a signature entry ```json {"alg":"ed25519","pk":"","sig":""} ``` - Keys alphabetical (enforced by recursive canonicalization). - `alg`: today always `"ed25519"`. Future algorithms add new literal values, not OIDs. - `pk`: hex-encoded public key bytes (32 bytes for Ed25519 -> 64 hex chars). - `sig`: hex-encoded signature bytes (64 bytes for Ed25519 -> 128 hex chars). `signatures` is an array so multi-signer is a pure extension -- just append additional entries. Verifier iterates and validates each independently. ### 3.2 The signing-view algorithm The core problem: if `row_hash` commits to the payload, and the payload contains the signature, the signature cannot sign the final bytes (it would have to contain itself). Solution -- sign a canonical view with `signatures` forced empty, then splice the signature in, re-canonicalize, and hash the spliced bytes. Both sides of the envelope commit to each other: **Sign (emission)**: 1. Start from caller's `PayloadFields`. 2. Force `payload_version = 2`, `signatures = []`. 3. Build the **signing view**: canonical bytes with `signatures=[]`. `payload::signing_view_bytes(&fields)`. 4. Signer produces `sig = sign(signing_view_bytes)`. 5. Splice `{alg, pk, sig}` (all hex-encoded) into `signatures`. 6. Re-canonicalize -- these are the **final stored bytes** (`raw_payload`). 7. `row_hash = SipHash-2-4(key, raw_payload)`. 8. `anchor_hash = SHA-256(row_hash || 0x00 || entity_id || 0x00 || chain_id)`. The row_hash therefore commits to the signature; the signature commits to everything *except* itself (via the signing view). **Verify (consumption)**: 1. Parse `raw_payload` back into `PayloadFields` (`chain::parse_raw_payload`). 2. If `payload_version < 2` or `signatures` is empty -> skip signature check. v1 rows and unsigned v2 rows pass the signature stage by design. 3. Rebuild the signing view by replacing `signatures` with `[]` and re-canonicalizing. 4. For each signature entry: - Dispatch on `alg`. Unknown -> `UnknownSignatureAlg { alg }`. - Hex-decode `pk` and `sig`. Decode failure -> `SignatureMalformed`. - `Ed25519Verifier::verify(&pk, signing_view_bytes, &sig)`. Failure -> `SignatureInvalid`. 5. `row_hash` is checked **independently** against the full stored bytes. Tampering with just the sig bytes fires `RowHashMismatch`; recomputing row_hash to hide the sig-tamper still fires `SignatureInvalid`. Attackers lose both ways. ### 3.3 Worked example (conceptual) Starting fields (abbreviated): ```json { "chain_id": "chain_acme_000001", "data": {"temp_c": 21}, "entity_id": "acme", "payload_version": 2, "prev_hash": "genesis", "sequence": 0, "signatures": [], ... } ``` Signing view = canonical bytes of that object exactly as shown (with all other canonical fields populated). Signer produces `sig` over those bytes. The final stored bytes: ```json { "chain_id": "chain_acme_000001", "data": {"temp_c": 21}, "entity_id": "acme", "payload_version": 2, "prev_hash": "genesis", "sequence": 0, "signatures": [{"alg":"ed25519","pk":"ab...cd","sig":"12...ef"}], ... } ``` `row_hash = siphash24(key, )`. This is what lands in the DB. ### 3.4 Nested object canonicalization (why it matters for v2) Signature entries are JSON objects. They MUST be canonicalized (keys sorted) before serialization, otherwise `serde_json::preserve_order` leaks caller insertion order into the canonical bytes and different emitters produce different `row_hash` for the same logical signature. `build_payload` explicitly canonicalizes the signatures array via the recursive `canonicalize` helper. See tag `s5_learn_preserve_order_requires_canonicalize_nested`. --- ## 4. VerifyError taxonomy `verify_links` returns `VerifyReport { ok, row_count, errors }`. Errors carry `{sequence, chain_id, kind}`; the kinds relevant to payload format: | Kind | Trigger | |------------------------|---------------------------------------------------------------------| | `PrevHashMismatch` | `link.prev_hash` != previous row's `row_hash` (or `genesis` on first row). | | `MissingRawPayload` | Non-BREAK row with no `raw_payload`. | | `InvalidRawPayload` | `raw_payload` does not parse as JSON or is missing required fields. | | `MissingRowHash` | Non-BREAK row with no `row_hash`. | | `RowHashMismatch` | Recomputed SipHash != stored `row_hash`. | | `AnchorHashMismatch` | Recomputed SHA-256 != stored `anchor_hash` (when stored). | | `SignatureInvalid` | v2: Ed25519 verify failed against signing view. | | `SignatureMalformed` | v2: signature entry not a JSON object, pk/sig hex decode failed. | | `UnknownSignatureAlg` | v2: `alg` field not recognized (only `ed25519` today). | BREAK markers carry no hashes and reset `prev` to `genesis` for the next segment. They are not counted in `row_count`. --- ## 5. API surface ### 5.1 `trusty-core` (pure library) ```rust use trusty_core::{ compute_record, compute_record_signed, Signer, Ed25519Signer, PayloadFields, SecretKey, verify_links, }; // v1 path -- unsigned. let rec = compute_record(&fields, &key)?; // rec.raw_payload has "payload_version":1, "signatures":[] // v2 path -- signed with any Signer impl. let signer = Ed25519Signer::generate(); let rec = compute_record_signed(&fields, &key, &signer)?; // rec.raw_payload has "payload_version":2 and one signatures entry. // Caller's fields.payload_version and fields.signatures are overridden. // Verify (version-agnostic). let report = verify_links(&entity_id, &links, &key); assert!(report.ok); ``` The `Signer` trait is byte-oriented: ```rust pub trait Signer { fn sign(&self, msg: &[u8]) -> Vec; fn public_key_bytes(&self) -> Vec; } ``` HSM or remote-signer integrations implement this trait directly; no ed25519-dalek dependency leaks into chain code. ### 5.2 `trusty-sqlite` (storage adapter) ```rust let store = Trusty::open(path, key)?; // v1 append let receipt = store.append(entity, data, source_url)?; // v2 append -- threads Option<&dyn Signer> through the append pipeline let receipt = store.append_signed(entity, data, source_url, &signer)?; // Custom extraction_method store.append_with_method(entity, data, source_url, method)?; store.append_with_method_signed(entity, data, source_url, method, &signer)?; ``` Mixed v1/v2 chains verify fine; dispatch is per-row on stored `payload_version`. You can flip signing on/off per-row at will. --- ## 6. Locked invariants - **v1 byte-shape is frozen.** Same field set, same canonical ordering, same `hash_alg` string, `signatures:[]`, `payload_version:1`. Tag: `s1_invariant_payload_v1_canonical`. - **v1 is preserved under v2.** v2 was shipped additive-only in S5 -- calling `compute_record` on a v1 `PayloadFields` emits exactly the same bytes it did before v2 existed. Tag: `s5_invariant_v1_byte_shape_preserved`. - **row_hash BE-hex encoding is canonical.** Decision `s2_decision_be_encoding_canonical`. - **Dual hash inputs are locked.** SipHash-2-4 keyed + SHA-256 unkeyed with null-byte domain separation. Tag: `s1_invariant_payload_v1_canonical`. - **Signature algorithm string is a literal.** No OID dance, no algorithm negotiation. Adding an algorithm = adding a new literal value in verify dispatch. - **row_hash commits to the signature bytes** (v2). Anything that would make the signature not affect `row_hash` is a regression. New algorithms, new fields, new versions must be purely additive and must leave v1 bytes byte-identical. Run `cargo test -p trusty-core` and inspect the `v1_*` tests before and after any payload-layer change.