316 lines
13 KiB
Markdown
316 lines
13 KiB
Markdown
# Trusty payload format -- v1 + v2
|
|
|
|
Canonical byte-level specification of what goes into `raw_payload`, what
|
|
`row_hash` commits to, and how v2 signatures are produced and verified.
|
|
|
|
This document is consensus-critical. v1 byte-shape is locked by the
|
|
invariant tests in `trusty-core` (see tag `s5_invariant_v1_byte_shape_preserved`
|
|
and the `v1_*` tests in `payload.rs` / `chain.rs`). v2 is purely additive
|
|
and must preserve v1 bytes exactly when `payload_version=1`. The original
|
|
Python port-source is archival only and no longer gates changes here.
|
|
|
|
Source of truth in-tree:
|
|
- `crates/trusty-core/src/payload.rs` -- canonical serializer, constants.
|
|
- `crates/trusty-core/src/chain.rs` -- `compute_record`, `compute_record_signed`,
|
|
`verify_links`, `verify_row_signatures`.
|
|
- `crates/trusty-core/src/hash.rs` -- `row_hash`, `anchor_hash`.
|
|
- `crates/trusty-core/src/crypto.rs` -- `Signer`, `Verifier`, Ed25519 impls.
|
|
|
|
---
|
|
|
|
## 1. Canonical form
|
|
|
|
Every `raw_payload` is a JSON object. Its canonical byte form is:
|
|
|
|
1. **Keys sorted alphabetically**, recursively (every nested object gets the
|
|
same treatment). Implemented via `BTreeMap` at the top level and a
|
|
recursive `canonicalize` on all nested `serde_json::Value::Object` nodes.
|
|
2. **No whitespace** -- `serde_json::to_string` (not `to_string_pretty`).
|
|
3. **UTF-8**. No BOM.
|
|
4. **Array order preserved**. Only object keys are sorted; arrays keep caller
|
|
order (this matters for `signatures`, which is an array -- but each entry
|
|
inside is itself an object and gets its keys sorted).
|
|
|
|
### 1.1 Field list (alphabetical -- the on-wire order)
|
|
|
|
| Field | Type | Notes |
|
|
|---------------------|------------|-----------------------------------------------------------------------|
|
|
| `chain_id` | string | `chain_<entity_id>_<6-digit-generation>` (e.g. `chain_acme_000001`). |
|
|
| `data` | any JSON | Caller-provided. Canonicalized recursively. Capped at 65,536 bytes after canonicalization. |
|
|
| `detected_at` | string | ISO-8601 UTC. Time trusty observed the event. |
|
|
| `entity_id` | string | Opaque caller identifier. |
|
|
| `extraction_method` | string | Caller tag for the provenance source (e.g. `trusty`, `scraper-v3`). |
|
|
| `hash_alg` | string | Always `"siphash24+sha256"` today. Verifiers read this before recomputing `row_hash`. |
|
|
| `payload_version` | number | `1` (v1) or `2` (v2). |
|
|
| `prev_hash` | string | Previous row's `row_hash` (16 hex chars), or the literal `"genesis"` on a segment's first row. |
|
|
| `received_at` | string | ISO-8601 UTC with sub-second precision. Distinct from `detected_at`; never collapse. |
|
|
| `sequence` | number | Per-chain monotonically increasing `i64`. Genesis = 0. |
|
|
| `signatures` | array | v1: always `[]`. v2: one object per signer (see Section 3). |
|
|
| `source_url` | string | Where the data came from. |
|
|
|
|
### 1.2 `data` size cap
|
|
|
|
`MAX_DATA_BYTES = 65_536` enforced after canonicalization. Exceeding it
|
|
raises `TrustyError::DataTooLarge`. Put blobs in object storage and reference
|
|
them by `{uri, sha256}` inside `data` -- do not inline them.
|
|
|
|
---
|
|
|
|
## 2. Dual hash
|
|
|
|
Each row carries two hashes. Both are computed in `trusty-core/src/hash.rs`
|
|
and are byte-identical to the Python reference.
|
|
|
|
### 2.1 `row_hash` -- SipHash-2-4
|
|
|
|
- Keyed with the 16-byte `SecretKey` (split into `k0` / `k1` as two
|
|
little-endian `u64`s, `bytes[0..8]` and `bytes[8..16]`).
|
|
- Input: the exact canonical payload bytes (`raw_payload` above).
|
|
- Output: 64-bit digest rendered as **lowercase hex**, **16 chars**, wire
|
|
representation derived from `format!("{:016x}", digest)`. This is a
|
|
big-endian-style hex rendering of the integer value. **BE hex is
|
|
canonical; any legacy LE rendering (e.g. earlier threadkeeper code) is
|
|
non-canonical and must be adapter-converted when crossing the boundary.**
|
|
See tag `s2_decision_be_encoding_canonical`.
|
|
|
|
### 2.2 `anchor_hash` -- SHA-256
|
|
|
|
Input byte layout:
|
|
```
|
|
row_hash_bytes || 0x00 || entity_id_bytes || 0x00 || chain_id_bytes
|
|
```
|
|
|
|
All three text fields are UTF-8. Output is lowercase hex, **64 chars**.
|
|
Null bytes are domain separators between fields -- do not drop them.
|
|
|
|
The dual-hash shape (keyed SipHash + keyless SHA-256 with null-byte domain
|
|
separation) is locked in `trusty-core/src/hash.rs` and covered by the
|
|
`anchor_hash_*` tests there. It is the only hash layout trusty recognises.
|
|
|
|
### 2.3 Why two hashes
|
|
|
|
SipHash-2-4 is a keyed fingerprint: cheap, fast, tamper-evident against
|
|
adversaries who do not hold the key. It is the chain-integrity primitive.
|
|
|
|
SHA-256 is the cryptographic anchor: public, collision-resistant, suitable
|
|
for external receipts and cross-system reference. The anchor is what you
|
|
publish or notarize; the row hash is what you use to link the chain.
|
|
|
|
Dual hash is non-negotiable -- do not collapse to one. See
|
|
`CLAUDE.md#design-rules`.
|
|
|
|
---
|
|
|
|
## 3. v2 signatures
|
|
|
|
v2 adds authenticated signatures over the payload. v1 is unchanged.
|
|
|
|
### 3.1 Shape of a signature entry
|
|
|
|
```json
|
|
{"alg":"ed25519","pk":"<hex>","sig":"<hex>"}
|
|
```
|
|
|
|
- Keys alphabetical (enforced by recursive canonicalization).
|
|
- `alg`: today always `"ed25519"`. Future algorithms add new literal values,
|
|
not OIDs.
|
|
- `pk`: hex-encoded public key bytes (32 bytes for Ed25519 -> 64 hex chars).
|
|
- `sig`: hex-encoded signature bytes (64 bytes for Ed25519 -> 128 hex chars).
|
|
|
|
`signatures` is an array so multi-signer is a pure extension -- just append
|
|
additional entries. Verifier iterates and validates each independently.
|
|
|
|
### 3.2 The signing-view algorithm
|
|
|
|
The core problem: if `row_hash` commits to the payload, and the payload
|
|
contains the signature, the signature cannot sign the final bytes (it would
|
|
have to contain itself).
|
|
|
|
Solution -- sign a canonical view with `signatures` forced empty, then
|
|
splice the signature in, re-canonicalize, and hash the spliced bytes. Both
|
|
sides of the envelope commit to each other:
|
|
|
|
**Sign (emission)**:
|
|
|
|
1. Start from caller's `PayloadFields`.
|
|
2. Force `payload_version = 2`, `signatures = []`.
|
|
3. Build the **signing view**: canonical bytes with `signatures=[]`.
|
|
`payload::signing_view_bytes(&fields)`.
|
|
4. Signer produces `sig = sign(signing_view_bytes)`.
|
|
5. Splice `{alg, pk, sig}` (all hex-encoded) into `signatures`.
|
|
6. Re-canonicalize -- these are the **final stored bytes** (`raw_payload`).
|
|
7. `row_hash = SipHash-2-4(key, raw_payload)`.
|
|
8. `anchor_hash = SHA-256(row_hash || 0x00 || entity_id || 0x00 || chain_id)`.
|
|
|
|
The row_hash therefore commits to the signature; the signature commits to
|
|
everything *except* itself (via the signing view).
|
|
|
|
**Verify (consumption)**:
|
|
|
|
1. Parse `raw_payload` back into `PayloadFields` (`chain::parse_raw_payload`).
|
|
2. If `payload_version < 2` or `signatures` is empty -> skip signature
|
|
check. v1 rows and unsigned v2 rows pass the signature stage by design.
|
|
3. Rebuild the signing view by replacing `signatures` with `[]` and
|
|
re-canonicalizing.
|
|
4. For each signature entry:
|
|
- Dispatch on `alg`. Unknown -> `UnknownSignatureAlg { alg }`.
|
|
- Hex-decode `pk` and `sig`. Decode failure -> `SignatureMalformed`.
|
|
- `Ed25519Verifier::verify(&pk, signing_view_bytes, &sig)`. Failure ->
|
|
`SignatureInvalid`.
|
|
5. `row_hash` is checked **independently** against the full stored bytes.
|
|
Tampering with just the sig bytes fires `RowHashMismatch`; recomputing
|
|
row_hash to hide the sig-tamper still fires `SignatureInvalid`. Attackers
|
|
lose both ways.
|
|
|
|
### 3.3 Worked example (conceptual)
|
|
|
|
Starting fields (abbreviated):
|
|
|
|
```json
|
|
{
|
|
"chain_id": "chain_acme_000001",
|
|
"data": {"temp_c": 21},
|
|
"entity_id": "acme",
|
|
"payload_version": 2,
|
|
"prev_hash": "genesis",
|
|
"sequence": 0,
|
|
"signatures": [],
|
|
...
|
|
}
|
|
```
|
|
|
|
Signing view = canonical bytes of that object exactly as shown (with all
|
|
other canonical fields populated).
|
|
|
|
Signer produces `sig` over those bytes. The final stored bytes:
|
|
|
|
```json
|
|
{
|
|
"chain_id": "chain_acme_000001",
|
|
"data": {"temp_c": 21},
|
|
"entity_id": "acme",
|
|
"payload_version": 2,
|
|
"prev_hash": "genesis",
|
|
"sequence": 0,
|
|
"signatures": [{"alg":"ed25519","pk":"ab...cd","sig":"12...ef"}],
|
|
...
|
|
}
|
|
```
|
|
|
|
`row_hash = siphash24(key, <final bytes>)`. This is what lands in the DB.
|
|
|
|
### 3.4 Nested object canonicalization (why it matters for v2)
|
|
|
|
Signature entries are JSON objects. They MUST be canonicalized (keys sorted)
|
|
before serialization, otherwise `serde_json::preserve_order` leaks caller
|
|
insertion order into the canonical bytes and different emitters produce
|
|
different `row_hash` for the same logical signature. `build_payload`
|
|
explicitly canonicalizes the signatures array via the recursive
|
|
`canonicalize` helper. See tag
|
|
`s5_learn_preserve_order_requires_canonicalize_nested`.
|
|
|
|
---
|
|
|
|
## 4. VerifyError taxonomy
|
|
|
|
`verify_links` returns `VerifyReport { ok, row_count, errors }`. Errors
|
|
carry `{sequence, chain_id, kind}`; the kinds relevant to payload format:
|
|
|
|
| Kind | Trigger |
|
|
|------------------------|---------------------------------------------------------------------|
|
|
| `PrevHashMismatch` | `link.prev_hash` != previous row's `row_hash` (or `genesis` on first row). |
|
|
| `MissingRawPayload` | Non-BREAK row with no `raw_payload`. |
|
|
| `InvalidRawPayload` | `raw_payload` does not parse as JSON or is missing required fields. |
|
|
| `MissingRowHash` | Non-BREAK row with no `row_hash`. |
|
|
| `RowHashMismatch` | Recomputed SipHash != stored `row_hash`. |
|
|
| `AnchorHashMismatch` | Recomputed SHA-256 != stored `anchor_hash` (when stored). |
|
|
| `SignatureInvalid` | v2: Ed25519 verify failed against signing view. |
|
|
| `SignatureMalformed` | v2: signature entry not a JSON object, pk/sig hex decode failed. |
|
|
| `UnknownSignatureAlg` | v2: `alg` field not recognized (only `ed25519` today). |
|
|
|
|
BREAK markers carry no hashes and reset `prev` to `genesis` for the next
|
|
segment. They are not counted in `row_count`.
|
|
|
|
---
|
|
|
|
## 5. API surface
|
|
|
|
### 5.1 `trusty-core` (pure library)
|
|
|
|
```rust
|
|
use trusty_core::{
|
|
compute_record, compute_record_signed,
|
|
Signer, Ed25519Signer,
|
|
PayloadFields, SecretKey,
|
|
verify_links,
|
|
};
|
|
|
|
// v1 path -- unsigned.
|
|
let rec = compute_record(&fields, &key)?;
|
|
// rec.raw_payload has "payload_version":1, "signatures":[]
|
|
|
|
// v2 path -- signed with any Signer impl.
|
|
let signer = Ed25519Signer::generate();
|
|
let rec = compute_record_signed(&fields, &key, &signer)?;
|
|
// rec.raw_payload has "payload_version":2 and one signatures entry.
|
|
// Caller's fields.payload_version and fields.signatures are overridden.
|
|
|
|
// Verify (version-agnostic).
|
|
let report = verify_links(&entity_id, &links, &key);
|
|
assert!(report.ok);
|
|
```
|
|
|
|
The `Signer` trait is byte-oriented:
|
|
|
|
```rust
|
|
pub trait Signer {
|
|
fn sign(&self, msg: &[u8]) -> Vec<u8>;
|
|
fn public_key_bytes(&self) -> Vec<u8>;
|
|
}
|
|
```
|
|
|
|
HSM or remote-signer integrations implement this trait directly; no
|
|
ed25519-dalek dependency leaks into chain code.
|
|
|
|
### 5.2 `trusty-sqlite` (storage adapter)
|
|
|
|
```rust
|
|
let store = Trusty::open(path, key)?;
|
|
|
|
// v1 append
|
|
let receipt = store.append(entity, data, source_url)?;
|
|
|
|
// v2 append -- threads Option<&dyn Signer> through the append pipeline
|
|
let receipt = store.append_signed(entity, data, source_url, &signer)?;
|
|
|
|
// Custom extraction_method
|
|
store.append_with_method(entity, data, source_url, method)?;
|
|
store.append_with_method_signed(entity, data, source_url, method, &signer)?;
|
|
```
|
|
|
|
Mixed v1/v2 chains verify fine; dispatch is per-row on stored
|
|
`payload_version`. You can flip signing on/off per-row at will.
|
|
|
|
---
|
|
|
|
## 6. Locked invariants
|
|
|
|
- **v1 byte-shape is frozen.** Same field set, same canonical ordering, same
|
|
`hash_alg` string, `signatures:[]`, `payload_version:1`. Tag:
|
|
`s1_invariant_payload_v1_canonical`.
|
|
- **v1 is preserved under v2.** v2 was shipped additive-only in S5 --
|
|
calling `compute_record` on a v1 `PayloadFields` emits exactly the same
|
|
bytes it did before v2 existed. Tag: `s5_invariant_v1_byte_shape_preserved`.
|
|
- **row_hash BE-hex encoding is canonical.** Decision `s2_decision_be_encoding_canonical`.
|
|
- **Dual hash inputs are locked.** SipHash-2-4 keyed + SHA-256 unkeyed with
|
|
null-byte domain separation. Tag: `s1_invariant_payload_v1_canonical`.
|
|
- **Signature algorithm string is a literal.** No OID dance, no algorithm
|
|
negotiation. Adding an algorithm = adding a new literal value in verify
|
|
dispatch.
|
|
- **row_hash commits to the signature bytes** (v2). Anything that would make
|
|
the signature not affect `row_hash` is a regression.
|
|
|
|
New algorithms, new fields, new versions must be purely additive and must
|
|
leave v1 bytes byte-identical. Run `cargo test -p trusty-core` and inspect
|
|
the `v1_*` tests before and after any payload-layer change.
|