Trusty/docs/payload-v2.md

Trusty payload format -- v1 + v2

Canonical byte-level specification of what goes into raw_payload, what row_hash commits to, and how v2 signatures are produced and verified.

This document is consensus-critical. v1 byte-shape is locked by the invariant tests in trusty-core (see tag s5_invariant_v1_byte_shape_preserved and the v1_* tests in payload.rs / chain.rs). v2 is purely additive and must preserve v1 bytes exactly when payload_version=1. The original Python port-source is archival only and no longer gates changes here.

Source of truth in-tree:

• crates/trusty-core/src/payload.rs -- canonical serializer, constants.
• crates/trusty-core/src/chain.rs -- compute_record, compute_record_signed, verify_links, verify_row_signatures.
• crates/trusty-core/src/hash.rs -- row_hash, anchor_hash.
• crates/trusty-core/src/crypto.rs -- Signer, Verifier, Ed25519 impls.

---

1. Canonical form

Every raw_payload is a JSON object. Its canonical byte form is:

1. Keys sorted alphabetically, recursively (every nested object gets the same treatment). Implemented via BTreeMap at the top level and a recursive canonicalize on all nested serde_json::Value::Object nodes.
2. No whitespace -- serde_json::to_string (not to_string_pretty).
3. UTF-8. No BOM.
4. Array order preserved. Only object keys are sorted; arrays keep caller order (this matters for signatures, which is an array -- but each entry inside is itself an object and gets its keys sorted).

1.1 Field list (alphabetical -- the on-wire order)

Field	Type	Notes
chain_id	string	chain_<entity_id>_<6-digit-generation> (e.g. chain_acme_000001).
data	any JSON	Caller-provided. Canonicalized recursively. Capped at 65,536 bytes after canonicalization.
detected_at	string	ISO-8601 UTC. Time trusty observed the event.
entity_id	string	Opaque caller identifier.
extraction_method	string	Caller tag for the provenance source (e.g. trusty, scraper-v3).
hash_alg	string	Always "siphash24+sha256" today. Verifiers read this before recomputing row_hash.
payload_version	number	1 (v1) or 2 (v2).
prev_hash	string	Previous row's row_hash (16 hex chars), or the literal "genesis" on a segment's first row.
received_at	string	ISO-8601 UTC with sub-second precision. Distinct from detected_at; never collapse.
sequence	number	Per-chain monotonically increasing i64. Genesis = 0.
signatures	array	v1: always []. v2: one object per signer (see Section 3).
source_url	string	Where the data came from.

1.2 data size cap

MAX_DATA_BYTES = 65_536 enforced after canonicalization. Exceeding it raises TrustyError::DataTooLarge. Put blobs in object storage and reference them by {uri, sha256} inside data -- do not inline them.

---

2. Dual hash

Each row carries two hashes. Both are computed in trusty-core/src/hash.rs and are byte-identical to the Python reference.

2.1 row_hash -- SipHash-2-4

• Keyed with the 16-byte SecretKey (split into k0 / k1 as two little-endian u64s, bytes[0..8] and bytes[8..16]).
• Input: the exact canonical payload bytes (raw_payload above).
• Output: 64-bit digest rendered as lowercase hex, 16 chars, wire representation derived from format!("{:016x}", digest). This is a big-endian-style hex rendering of the integer value. BE hex is canonical; any legacy LE rendering (e.g. earlier threadkeeper code) is non-canonical and must be adapter-converted when crossing the boundary. See tag s2_decision_be_encoding_canonical.

2.2 anchor_hash -- SHA-256

Input byte layout:

row_hash_bytes || 0x00 || entity_id_bytes || 0x00 || chain_id_bytes

All three text fields are UTF-8. Output is lowercase hex, 64 chars. Null bytes are domain separators between fields -- do not drop them.

The dual-hash shape (keyed SipHash + keyless SHA-256 with null-byte domain separation) is locked in trusty-core/src/hash.rs and covered by the anchor_hash_* tests there. It is the only hash layout trusty recognises.

2.3 Why two hashes

SipHash-2-4 is a keyed fingerprint: cheap, fast, tamper-evident against adversaries who do not hold the key. It is the chain-integrity primitive.

SHA-256 is the cryptographic anchor: public, collision-resistant, suitable for external receipts and cross-system reference. The anchor is what you publish or notarize; the row hash is what you use to link the chain.

Dual hash is non-negotiable -- do not collapse to one. See CLAUDE.md#design-rules.

---

3. v2 signatures

v2 adds authenticated signatures over the payload. v1 is unchanged.

3.1 Shape of a signature entry

{"alg":"ed25519","pk":"<hex>","sig":"<hex>"}

• Keys alphabetical (enforced by recursive canonicalization).
• alg: today always "ed25519". Future algorithms add new literal values, not OIDs.
• pk: hex-encoded public key bytes (32 bytes for Ed25519 -> 64 hex chars).
• sig: hex-encoded signature bytes (64 bytes for Ed25519 -> 128 hex chars).

signatures is an array so multi-signer is a pure extension -- just append additional entries. Verifier iterates and validates each independently.

3.2 The signing-view algorithm

The core problem: if row_hash commits to the payload, and the payload contains the signature, the signature cannot sign the final bytes (it would have to contain itself).

Solution -- sign a canonical view with signatures forced empty, then splice the signature in, re-canonicalize, and hash the spliced bytes. Both sides of the envelope commit to each other:

Sign (emission):

1. Start from caller's PayloadFields.
2. Force payload_version = 2, signatures = [].
3. Build the signing view: canonical bytes with signatures=[]. payload::signing_view_bytes(&fields).
4. Signer produces sig = sign(signing_view_bytes).
5. Splice {alg, pk, sig} (all hex-encoded) into signatures.
6. Re-canonicalize -- these are the final stored bytes (raw_payload).
7. row_hash = SipHash-2-4(key, raw_payload).
8. anchor_hash = SHA-256(row_hash || 0x00 || entity_id || 0x00 || chain_id).

The row_hash therefore commits to the signature; the signature commits to everything except itself (via the signing view).

Verify (consumption):

1. Parse raw_payload back into PayloadFields (chain::parse_raw_payload).
2. If payload_version < 2 or signatures is empty -> skip signature check. v1 rows and unsigned v2 rows pass the signature stage by design.
3. Rebuild the signing view by replacing signatures with [] and re-canonicalizing.
4. For each signature entry:
   • Dispatch on alg. Unknown -> UnknownSignatureAlg { alg }.
   • Hex-decode pk and sig. Decode failure -> SignatureMalformed.
   • Ed25519Verifier::verify(&pk, signing_view_bytes, &sig). Failure -> SignatureInvalid.
5. row_hash is checked independently against the full stored bytes. Tampering with just the sig bytes fires RowHashMismatch; recomputing row_hash to hide the sig-tamper still fires SignatureInvalid. Attackers lose both ways.

3.3 Worked example (conceptual)

Starting fields (abbreviated):

{
  "chain_id": "chain_acme_000001",
  "data": {"temp_c": 21},
  "entity_id": "acme",
  "payload_version": 2,
  "prev_hash": "genesis",
  "sequence": 0,
  "signatures": [],
  ...
}

Signing view = canonical bytes of that object exactly as shown (with all other canonical fields populated).

Signer produces sig over those bytes. The final stored bytes:

{
  "chain_id": "chain_acme_000001",
  "data": {"temp_c": 21},
  "entity_id": "acme",
  "payload_version": 2,
  "prev_hash": "genesis",
  "sequence": 0,
  "signatures": [{"alg":"ed25519","pk":"ab...cd","sig":"12...ef"}],
  ...
}

row_hash = siphash24(key, <final bytes>). This is what lands in the DB.

3.4 Nested object canonicalization (why it matters for v2)

Signature entries are JSON objects. They MUST be canonicalized (keys sorted) before serialization, otherwise serde_json::preserve_order leaks caller insertion order into the canonical bytes and different emitters produce different row_hash for the same logical signature. build_payload explicitly canonicalizes the signatures array via the recursive canonicalize helper. See tag s5_learn_preserve_order_requires_canonicalize_nested.

---

4. VerifyError taxonomy

verify_links returns VerifyReport { ok, row_count, errors }. Errors carry {sequence, chain_id, kind}; the kinds relevant to payload format:

Kind	Trigger
PrevHashMismatch	link.prev_hash != previous row's row_hash (or genesis on first row).
MissingRawPayload	Non-BREAK row with no raw_payload.
InvalidRawPayload	raw_payload does not parse as JSON or is missing required fields.
MissingRowHash	Non-BREAK row with no row_hash.
RowHashMismatch	Recomputed SipHash != stored row_hash.
AnchorHashMismatch	Recomputed SHA-256 != stored anchor_hash (when stored).
SignatureInvalid	v2: Ed25519 verify failed against signing view.
SignatureMalformed	v2: signature entry not a JSON object, pk/sig hex decode failed.
UnknownSignatureAlg	v2: alg field not recognized (only ed25519 today).

BREAK markers carry no hashes and reset prev to genesis for the next segment. They are not counted in row_count.

---

5. API surface

5.1 trusty-core (pure library)

use trusty_core::{
    compute_record, compute_record_signed,
    Signer, Ed25519Signer,
    PayloadFields, SecretKey,
    verify_links,
};

// v1 path -- unsigned.
let rec = compute_record(&fields, &key)?;
// rec.raw_payload has "payload_version":1, "signatures":[]

// v2 path -- signed with any Signer impl.
let signer = Ed25519Signer::generate();
let rec = compute_record_signed(&fields, &key, &signer)?;
// rec.raw_payload has "payload_version":2 and one signatures entry.
// Caller's fields.payload_version and fields.signatures are overridden.

// Verify (version-agnostic).
let report = verify_links(&entity_id, &links, &key);
assert!(report.ok);

The Signer trait is byte-oriented:

pub trait Signer {
    fn sign(&self, msg: &[u8]) -> Vec<u8>;
    fn public_key_bytes(&self) -> Vec<u8>;
}

HSM or remote-signer integrations implement this trait directly; no ed25519-dalek dependency leaks into chain code.

5.2 trusty-sqlite (storage adapter)

let store = Trusty::open(path, key)?;

// v1 append
let receipt = store.append(entity, data, source_url)?;

// v2 append -- threads Option<&dyn Signer> through the append pipeline
let receipt = store.append_signed(entity, data, source_url, &signer)?;

// Custom extraction_method
store.append_with_method(entity, data, source_url, method)?;
store.append_with_method_signed(entity, data, source_url, method, &signer)?;

Mixed v1/v2 chains verify fine; dispatch is per-row on stored payload_version. You can flip signing on/off per-row at will.

---

6. Locked invariants

• v1 byte-shape is frozen. Same field set, same canonical ordering, same hash_alg string, signatures:[], payload_version:1. Tag: s1_invariant_payload_v1_canonical.
• v1 is preserved under v2. v2 was shipped additive-only in S5 -- calling compute_record on a v1 PayloadFields emits exactly the same bytes it did before v2 existed. Tag: s5_invariant_v1_byte_shape_preserved.
• row_hash BE-hex encoding is canonical. Decision s2_decision_be_encoding_canonical.
• Dual hash inputs are locked. SipHash-2-4 keyed + SHA-256 unkeyed with null-byte domain separation. Tag: s1_invariant_payload_v1_canonical.
• Signature algorithm string is a literal. No OID dance, no algorithm negotiation. Adding an algorithm = adding a new literal value in verify dispatch.
• row_hash commits to the signature bytes (v2). Anything that would make the signature not affect row_hash is a regression.

New algorithms, new fields, new versions must be purely additive and must leave v1 bytes byte-identical. Run cargo test -p trusty-core and inspect the v1_* tests before and after any payload-layer change.