Thirteen candidate stacks, one test bench, forty-one promises. Four findings that matter for secure data storage, group spaces and delivery, with the evidence behind each.
The test bench runs the same scenarios against every candidate: group membership under conflict (S1 to S11), an adversarial carrier (T1 to T12) and, since this week, pair delivery (T9, T10, Z1 to Z5). A candidate is a stack behind a narrow driver interface. Results are promises per port, not scores: each cell says whether all scenarios mapped to a promise passed. Everything below is reproducible from the public repository, linked at the end.
We run our authority log (membership decided by a conflict matrix, signed entries, policy) over two group key protocols: BeeKEM, Keyhive's CGKA, and MLS (RFC 9420, OpenMLS 0.9.1). Both deliver an identical row in the matrix: 25 promises kept, 1 not kept, 5 not applicable, 10 untested. The one gap is the same in both: a blind carrier that freezes a device's feed is not detectable without an attested target.
What MLS needs from the layer above, because it is linear (one commit per epoch): a single healer per group, reconciliation after every commit, a welcome that carries the digest of its admitting commit, and re-sealing of frames whose epoch lost a tiebreak. A room model where the commit itself decides membership keeps fewer promises on the same bench (see the two MLS room rows in the summary below; in the any-staged mode S4f and T2 hold only in some branches of the hash tie-break; the bench runs every branch, so both count as not kept). Candidates rltp-beekem, rltp-mls.
If a newcomer only receives keys from the epoch it joined, it cannot read earlier entries. That part is expected. What the bench showed in addition: the newcomer cannot integrate new entries either, because CRDT updates reference the entry before them. A second device of an existing member reads nothing at all.
For independent chat lines this never surfaces. For shared state (a map, a calendar, a board) a key chain across epochs is a precondition, not a comfort. Scenarios S5, S6, S6b on the model without history; the same scenarios pass with a predecessor table in every sealed frame.
We sent the same sealed envelope through four wires: our plain envelope, TSP Rev 3 and DIDComm v2 in authcrypt and anoncrypt mode. All four produce the same delivery behaviour: a sender status trias, at-least-once with receiver dedup, a pre-transport report when offline, carrier switch without losing the relationship. The wires differ only in what the carrier learns and where a foreign sender is rejected.
| Wire | Carrier sees sender | Frame size | Foreign sender rejected |
|---|---|---|---|
| plain envelope (rkid) | no | 668 B | at the document (proof) |
| TSP Rev 3 direct | yes, relationship VID | 996 B | at the wire (outer signature) |
| DIDComm v2 authcrypt | yes, skid | 2562 B | at the wire (ECDH-1PU) |
| DIDComm v2 anoncrypt | no | 2209 B | at the document (proof) |
A port must not rely on the wire's sender check; the document proof is the one that is always there. Identities per relationship on all wires. Libraries: @openvtc/vti-tsp-js 0.3.0, @openvtc/vti-didcomm-js 0.12.0.
One space per group, one collection of sealed frames, reading over the per-device changes feed, the space controller is a group key. Every scenario passes as it does over our own relay, including the adversarial carrier scenarios and a server switch by export and import.
What the server learns: the space id; per resource the sender device id and the frame label as plaintext fields, a resource id made of that device id and a sequence number, size and timestamp; the addressee of welcomes. Content, person names and the group secrets stay sealed. A pseudonym per device and group would remove the device ids from the server; that is not built. The gap: a frozen feed is not detectable while the server issues no attested target. That is exactly the job a group controller profile would have. Load at 30 members, 100 operations: about 290 ms per operation, half of it HTTP. Candidate rltp-beekem-pws.
Promises kept, partial, not kept and not applicable per candidate, counted from the cells of the full matrix, which shows every cell with its scenarios and what the bench recorded. Ten of the 41 promises have no scenario yet and are untested for every candidate; they are left out here.
| Candidate | kept | partial | not kept | n/a |
|---|---|---|---|---|
| RLTP over BeeKEM | 25 | 0 | 1 | 5 |
| RLTP over MLS | 25 | 0 | 1 | 5 |
| RLTP over BeeKEM or MLS, with a view-class service | 26 | 0 | 3 | 2 |
| RLTP over BeeKEM on Portable Web Spaces | 22 | 3 | 1 | 5 |
| RLTP with a log-replica service | 22 | 0 | 7 | 2 |
| MLS room, any-staged | 13 | 0 | 10 | 8 |
| MLS room, owner | 14 | 0 | 9 | 8 |
| Keyhive 0.3.0-alpha.1 | 12 | 2 | 6 | 11 |
| p2panda-spaces 0.7.1 | 10 | 2 | 6 | 13 |
| Keyhive over ARK with Subduction | 7 | 2 | 6 | 16 |
| WoT generation 2 (our own, in production) | 7 | 1 | 5 | 18 |
A promise is kept only if every scenario mapped to it passed. "Partial" means some scenarios passed and the rest were not applicable; it is not a kept promise. "Not applicable" means the candidate has no capability for the scenario (no policy, no service, no own transport) or the carrier fault never touched a frame of that kind. It is a gap in coverage, never a pass. RLTP over PWS is not the same row as RLTP over BeeKEM: three promises are partial and one is not applicable because the bench cannot watch traffic at the server (S5b, S7, S11), and the carrier switch (I15) is kept only there.
Welcome size, measured. With the ratchet tree in the welcome: 839 B at one member, 2.7 KB at ten, 6.5 KB at thirty, about 190 B per further member. A 16 KB message bound is reached around eighty members.