MOQtail LogoMOQtail

All posts

Two Gold Medals at IETF 127: What We Shipped Before the Interop

From 56/76 to 76/76 in a month: the relay observability layer, MCP server, performance work and interop fixes behind MOQtail's two gold medals at the IETF 127 hackathon.

Zafer Gurel

Zafer Gurel

@zafergurel

Two Gold Medals at IETF 127: What We Shipped Before the Interop

In October, at the MOQ interim, the draft-22 conformance suite gave the MOQtail relay 56 out of 76 on raw QUIC and 58 out of 76 on WebTransport. Not bad, not good.

A month later, at the IETF 127 hackathon on the 14th and 15th of November, the same suite gave it 76 out of 76 on both transports. On the same weekend, the end-to-end interop report awarded MOQtail a gold medal for draft-22.

Ali C. Begen was at the hackathon in San Francisco, while Deniz and I followed along from our homes in the #moq channel on Slack.

Here is what we shipped, what broke, and what's next.

The conformance matrix: 76/76 on QUIC and WebTransport The draft-22 conformance run, Saturday afternoon.

The two medals

The first came from the conformance suite that runs against every live relay listed in the moq-interop-runner. It covers ten areas, from forwarding preferences to joining FETCH, in 76 tests per transport. Here is where we started and where we finished:

Section Oct (QUIC) Oct (WT) Nov (QUIC) Nov (WT)
1. Basic Forwarding Preferences ❌ ❌ ✅ ✅
2. Object and Group Counts ❌ ❌ ✅ ✅
3. Object Sizes ❌ ❌ ✅ ✅
4. Group and Object Increments ✅ ✅ ✅ ✅
5. End of Group Markers ❌ ❌ ✅ ✅
6. Extensions ❌ ❌ ✅ ✅
7. Complex Scenarios ❌ ✅ ✅ ✅
8. PUBLISH ✅ ✅ ✅ ✅
9. FETCH Ranges ❌ ❌ ✅ ✅
10. Joining FETCH ❌ ✅ ✅ ✅
Total 56/76 58/76 76/76 76/76

The second came from the end-to-end interop report, which drives real browser sessions (setup, announcements, publish, subscribe, namespace discovery, updates and unsubscribe, over streams and datagrams) through each relay. MOQtail passed all 97 draft-22 tests and got the gold medal and the little celebration banner that comes with it.

The end-to-end interop report's result card for moqtail on draft-22: Full interop medal, 97 passed, 0 failed Yes, you have to click it to see the banner.

Neither result came from a single big fix. They came from a long tail of small ones, and almost every one of them was found with the tools below.

Seeing inside the relay

The headline feature of this release is an observability layer for the relay (#477). It was built for interop week, but it's the part we expect people to keep using.

  • An HTTP API under /api/v1: health and readiness, connections (including recently closed ones and why they closed), tracks, namespaces, upstreams, cluster state, and object processing latency as percentiles per relay, track or subscription. There's a slowlog and a Prometheus /metrics endpoint too. It runs on its own single-threaded runtime, so it never takes time from delivery. Measured against a build without it, it makes no difference to throughput, latency or CPU.
  • A web dashboard on top of that API, for looking at tracks and namespaces, following each connection's control messages, and downloading its qlog file.
  • qlog traces for every control message and every data stream header, in both the relay and the Rust client (#473, #474). Each control message is recorded with its decoded fields and its raw bytes in hex. When a message fails to parse, the raw bytes are recorded as an error event. With qlog off, it costs one atomic load per message. A small script, scripts/qlog-summary.py, turns a folder of traces into a readable summary of each peer: its SETUP, every request with its responses, and anything that failed to parse.
  • An MCP server, so an AI agent can ask the relay questions directly.

The relay dashboard's overview page: 10 connections, 3 tracks, 217 objects/s out, and charts of objects per second, throughput and p50/p99 processing time The dashboard's overview page.

A subscriber's connection page in the dashboard: QUIC stats, two track subscriptions, and the message timeline with SUBSCRIBE, SUBSCRIBE_OK, REQUEST_UPDATE and REQUEST_OK A connection's timeline: each request next to its response, with the raw bytes one click away.

The MCP server is the piece we didn't expect to like this much. Instead of clicking through pages, we can ask a coding agent what the production relay is doing. While writing this post, we asked it for an overview and got this back (trimmed):

> How is the production relay doing?
 
relay_overview →
  version:       moqtail/0.18.0
  connections:   14 active
  tracks:        22, with 1,138 cached groups
  objects in:    3,439,528  (11.2 GB)
  objects out:   2,781,790  (10.7 GB)
  dropped:       0
  subscribe/downstream: 4,102 ok

Then we asked why a recent connection's SUBSCRIBEs failed. The agent called connection_timeline and found four of them, each answered within two milliseconds with REQUEST_ERROR 16, "Unknown track namespace": no publisher had announced the namespace the subscriber asked for. It quoted the bytes of each request and response. That loop, from "something failed" to "here's the exact message", used to take a few minutes of log grepping. Now it takes one question.

Bugs, on both sides

Interop is where you find out which parts of your reading of the draft are only your reading. This round fixed a long list on our side:

  • FETCH: we now report End Of Track for cached FETCHes of a track that has ended (#481) and use the priority the subscriber asked for (#472).
  • Fills: a fill opened by REQUEST_UPDATE carried the subscription's Request ID instead of its own, so the client couldn't match the stream (#483).
  • Malformed input: an unparsable message on a request stream now closes the session with PROTOCOL_VIOLATION instead of being silently dropped (#483). A Mandatory Track Property on an Object now marks the track as malformed (#491).
  • Padding streams were treated as a bad header and stopped. Now they're read and discarded (#483).
  • NEW_GROUP_REQUEST is now passed upstream instead of being lost at the relay (#483).
  • Track cache: MAX_CACHE_DURATION is now honored (#487).
  • TLS: the relay accepts X25519MLKEM768 key shares, the post-quantum key exchange that recent browsers offer first (#489).
  • TypeScript client: objects of tracks that the peer pushes with PUBLISH are now delivered (#488).

It went the other way too, and the qlog traces made those conversations short, because they show exactly which bytes don't line up. TRACK_NAMESPACE_PREFIX is a good example. The draft encodes it as a plain track namespace: a field count followed by the fields, with no length in front. Our relay used to expect a length there, so one team's browser tests, which sent the correct form, failed against it, and we fixed that (#482). Two days later, another implementation updated a namespace prefix and the relay rejected it. The trace showed why: it was sending the length we had just stopped expecting, and the relay read that length, 16, as the number of fields and ran out of bytes. Its author checked the draft, agreed it had always been that way, and offered to fix it.

Our favorite example is FILL_PARAMETERS. Implementations didn't agree on whether it carries a length, a parameter count, or both. After a long thread in the working group's chat, everyone settled on length plus count for draft-22. Since then, the MOQtail relay also accepts the count-less form (#485), because being strict about something the group only just agreed on helps nobody.

Running it for real

A few changes were about running the relay in production rather than passing tests:

  • A faster hot path (#462). Payloads go to QUIC without copying, the receive buffer is no longer deep-copied on every read, and each Object is shared between subscribers instead of cloned per subscriber. Relay CPU seconds over 15 s on a 12-core Linux box:

    Change Load Before After
    Receive buffer not copied 15 tracks × 200 KB objects 18.09 10.05 (−44%)
    Send and receive copies removed 5 subscribers × 200 KB 7.71 5.94 (−23%)
    8 MiB socket buffers 1000 subscribers × 1.2 KB 83–96 38–40

    Slow subscribers can no longer grow memory without bound: there's a per-subscriber queue cap and an optional delivery timeout.

  • A config file, from Deniz Ugur (#466). Settings live in a YAML file, and durations take units (10s, 500ms).

  • Clustering, also Deniz's work (#467). Relays can form a mesh, and a publisher on one relay is reachable from any other. Each hop adds itself to the path and its cost to the route, and each subscription goes to the cheapest source without loops:

    hop_id: 1
    relay_cost: 1
    peers: [moqt://relay-b:4433]
  • GOAWAY (#476). Our clients used to ignore it until the relay timed them out. Now the Rust client and the TypeScript player follow a GOAWAY to the new relay it names, so a relay can be drained and its clients moved to another one without a reload.

The player

The player is Deniz Ugur's work, and it's one of the nicest things in this release. He built <moqtail-video>, a web component that brings MOQ playback to a page as a single HTML element, on its own or inside Video.js v10. Around it he added an ABR controller (#480), LOCMAF packaging (#479), MSF draft-02 catalogs, and live seeking limited to the window the relay can actually serve (#492, #493). It deserves its own post, so this is just a pointer to the README.

What's next

A release, very soon. Everything above ships in the next version of the relay and the moqtail npm package.

Straight to draft-24. We're skipping draft-23 and moving to draft-24, because we'd rather follow the interop targets than chase every intermediate draft. Expect it in a few weeks.

Try it

Thanks to everyone who ran their clients against our relay and told us when something was wrong. That's how we got to 76.