adsvc

Catalogue, index, config and sync

How adsvc stores ads and file maps, indexes them for matching, shares them between nodes, and how its config and web page work.

The catalogue file can be handed to anyone, so it holds no secrets, file names or URLs (the one exception is the local-only reports table, §3). Files are known only by ih:<infohash>/<idx>, c:<content key> or id:<hash> keys (internal/filekey), and ads by a content id plus a label.

1. Overview

            records (signed, replicated)
                    │ materialize
                    ▼
   ┌──────────── catalogue.db (SQLite, shareable) ─────────────┐
   │ records · ads · labels · votes · dups · file maps          │
   │ local-only: origin_policy · peers · pushers · pins ·       │
   │             reports · meta                                 │
   └────────────────────────────────────────────────────────────┘
                    │ select tracking set (trust, pins, ≤ max_ads)
                    ▼
        tracking.csr (immutable, mmapped)  +  overlay (in-memory map
                    │                          index, ads added since
                    ▼                          the last build)
        library.Library: Match / Get  ──►  detect.Scan / Pick

2. Identity and records (internal/record)

Node identity

Ad id

Record envelope

{kind, origin, seq, ts, body, sig}
Field Meaning
origin the author’s public key
seq per origin, monotonic
ts milliseconds, from a hybrid logical clock (last-writer-wins stays stable against clock skew)
body JSON bytes, stored verbatim
sig ed25519 over a domain prefix and every field; the record hash is SHA-256 of the same bytes

Kinds

All durations and positions are integer milliseconds.

Kind Body Semantics
ad.add {id, fp_version, duration_ms, points, type?: "ad"\|"intro", source?: {file_key, start_ms, end_ms}, supersedes?} adds a fingerprint; source says where it was enrolled from
ad.label {ad, label, type?} last writer wins by (ts, origin); this node’s own label always wins locally. type (ad or intro) changes the ad’s type the same way; a label without one leaves it alone
ad.vote {ad, value: -1\|+1, reason: good\|not_ad\|boundary\|dup, file_key?, start_ms?} only the latest vote per (ad, origin) counts
ad.dup {ad, canonical} near-duplicate link
ad.retract {ad} valid only from the ad’s author
file.map {key, aliases, size, duration_ms, analyzed_ms, ads: [{ad, start_ms, end_ms, score, confirmed}]} replaces this origin’s map for key

Points encoding

3. SQLite catalogue (internal/catalog)

Driver and connections

Migrations

Tables

All times are integer milliseconds.

Replicated (in snapshots):

Table Contents
records id (the local receive sequence peers pull by), hash, origin, seq, kind, ts, body, sig, via (the node it came from; NULL = written here), received

Materialized from the records:

Table Contents
ads id, fp_version, label, type (ad/intro, as added), type_eff, duration_ms, n_points, points, created, source_key/source_start_ms/source_end_ms, author, author_ts, record
ad_labels per (ad, origin): the latest label
ad_types per (ad, origin): the latest type that origin named (ad.label with a type); ads.type_eff is this node’s own, else the latest of any origin, else the one the ad was added with
votes per (ad, origin): value, reason, file_key, start_ms
dups per (ad, origin): canonical
retractions (ad, origin)
ad_records the records about each ad, for its history
file_maps, file_aliases, detections this node’s own file maps
peer_maps, peer_aliases, peer_detections other origins’ file maps

Local-only (local opinion and state, dropped from snapshots):

Table Contents
origin_policy per origin: name, weight, blocked, managed (set from the config file)
peers per configured peer name: node_id, pull and push cursors, last success or error, received, rejections by reason
pushers per node that pushed: last push, received, duplicates, rejections by reason
pins ads always in the tracking set
published_maps digest of the last file.map published per key
meta e.g. the digest of the last CSR build
reports ads a user saw, to mark later (§8). The one table with URLs, private to its user and the admins. Share a catalogue only as a snapshot (adsvc export, /sync/snapshot), never as the raw catalogue.db

Peer URLs and credentials live in the config file, not here.

4. Trust and tracking selection

Trust is local policy: computed on each node, never replicated.

trust(ad) = w(author)                                   # implicit +1 from the author
          + Σ_origin w(origin) · vote(origin, ad)
          + Σ_origin 0.5 · w(origin) · [origin has a confirmed detection of ad]
Origin Weight w
this node trust.self (default 10)
unknown origins trust.unknown (default 0.2)
listed origins trust.origins, or set on the Origins page
blocked origins 0; their records are dropped on import and never relayed

Tracking set

Active ads of the current fp_version, pinned ones first, then those with trust ≥ trust.min_trust, most trusted first, at most tracking.max_ads. Duplicates are left out: the canonical ad matches for them.

File maps at play time

When a session starts, its keys (primary key and aliases) are looked up and every origin’s detections are merged:

  1. Each detection’s ad is mapped to its canonical id, and detections are clustered by (canonical ad, start within ±0.25 s).
  2. A cluster is accepted if its ad is trusted and the summed origin weights reach trust.file_map_min. It is confirmed if any accepted origin confirmed it.
  3. Coverage (analyzed) comes only from this node’s own map: a peer’s claim that a range has no ads does not stop local analysis.

Other nodes’ detections are shown, never stored in or republished from this node’s own maps.

Who produces trash

The Origins and Peers pages (catalog.Origins, catalog.Peers) show:

The Files page flags a peer’s detection in a range this node analysed without finding it.

5. CSR tracking index (internal/library)

Layout

A posting is one landmark of one ad, packed in a u32 as dense ad:18 | frame:14. Hash lookup is two-level:

A flat 2^24 × u32 table would cost 64 MB whatever the library size; the two-level form costs 5 bytes per distinct hash plus 256 KB.

Limits: 18 bits for the ad (library.MaxAds = 262,144) and 14 bits for the frame (library.MaxDuration = 524 s per ad; Add returns ErrTooLong beyond that).

Voting

Votes go into a pooled dense tally: one run of u16 cells per ad, one cell per offset the query can align it at, with a guard cell at each end. Only touched cells are read and reset. It takes about 27 MB per concurrent Match at 10k ads; an index too big for 2^28 cells falls back to map voting.

Measured

Synthetic landmarks shaped like real ones (dt and frequency skew from the ads of the Babylon bench). 10k ads of 20–45 s is about 61M postings. Apple M3 Pro, go test -bench . ./internal/library:

  map index CSR, map votes CSR + dense tally
index heap, 10k ads 781 MiB 237.5 MiB 237.5 MiB
build, 10k ads 4.08 s 0.42 s 0.42 s
Match, 10 s block of programme audio 10.3 ms 9.8 ms 2.29 ms
Match, 10 s block inside an ad 21.4 ms 20.8 ms 4.80 ms
Match allocations 9–18 MiB same ~0 (pooled)

Built from the catalogue into the mapped file: 1.23 s and 490 MiB allocated; the open index keeps 1.3 MiB of heap, and the file is about 290 MB of page cache.

Overlay, tombstones, rebuild

File

Little-endian, sections aligned to 8 bytes:

header    magic "ADSVCSR1", format, fp_version, hash_bits (=24), low_bits (=8), n_ads,
          n_distinct, n_postings, set_digest [32]B, built_at   (80 bytes)
ids       n_ads × 16 B        sizes  n_ads × u32        start  (n_ads + 1) × u32
top       (2^16 + 1) × u32    post   n_postings × u32
low       n_distinct × u8     off    (n_distinct + 1) × u32

6. Config and SIGHUP (internal/config)

Every key is documented in config.yml.example.

SIGHUP, or the Config page’s Reload button, re-reads and validates the whole file. If it is invalid, the old config stays and a warning is logged. Otherwise:

Takes effect Keys
at once auth.*, upstream.*, detect.max_stall, detect.idle_timeout, detect.confirm_score, log.*, trust.*, tracking.*, sync.*
for new sessions detect.min_score, detect.mark_window, ffmpeg
after a restart (logged) listen, data_dir, media.*

trust.origins rows are re-seeded on every reload and are read-only on the Origins page.

7. Federated sync (internal/replica)

Every node serves and consumes the same endpoints. There is no hub.

Endpoint Purpose
GET /sync/info?nonce= {node_id, name, fp_version, schema, head}, signed over the caller’s nonce, so a relay cannot pass itself off as another node
GET /sync/log?after=<id> accepted records in receive order, as gzip-compressed record frames; never from blocked origins
GET /sync/ad/{id} the ad.add record of one ad
GET /sync/snapshot a SQLite file with only the records and the node id
POST /sync/push records from a node that cannot be pulled from (behind NAT)

Import checks

Every record passes these checks in order. Each rejection is counted against the peer (peers.rejected) or the pusher (pushers.rejected).

  1. The envelope is well-formed; a record already held is a duplicate, not a rejection.
  2. The signature is valid.
  3. The origin is not blocked.
  4. The origin is within its quota (trust.quota_per_day).
  5. The body is valid for its kind: for ad.add, the id matches the points and the duration and points are within limits; ad.retract only from the author; file.map keys only in ih:/c:/id: form.

Records for an unknown fp_version are kept and relayed but never tracked.

8. Web page (internal/admin, assets in web/)

Routing

At the root of the proxy’s listener. frontHandler gives the proxy its own routes (proxy.APIPath: /s, /healthz, /ads/*), /sync/* to the replica, and everything else to the page.

Access

Reports

A player that shows an ad it has no way to skip reports it: POST /ads/report?u=<source>[&id=|&ih=&idx=]&position=MS[&note=]. The Reports page lists the user’s reports (an admin sees everyone’s) by file, each with an mpv command (mpv --start=<t−20> '<base>/s?u=…', where base is public_url or the request’s host), and hands out adskip.lua with the user’s token written in. While a file has an open report, its session decodes everything, including ranges analysed before, so /ads/mark finds the audio. A successful /ads/mark deletes the user’s reports of that file within the ad ±15 s.

Pages

Page Contents
Reports (every user) reported ads by file, each with its mpv command and a delete button; the adskip.lua download and how to mark
Ads search by label or id; filters: origin, type, pinned, fp version; newest first or by label, 100 per page (filtered, sorted and paged in SQL, on stored columns only); trust, votes and state shown for the page’s rows; bulk vote, pin or delete; a dot for likely duplicates
Ad edit the label and type (ad.label); vote with a reason (ad.vote); pin; mark as a duplicate (ad.dup); delete (ad.retract for this node’s own ad, a −1 not_ad vote for another’s); its records (author, via, time); the files it was found in; quality: the weakest self score and the strongest other tracked ad per window of detect’s block, against detect.min_score
Files the list, most recently updated first, 100 per page; by key: the merged map, per-origin detections side by side, disagreements highlighted, this node’s own coverage
Origins the trash statistics of §4; set a weight or block; config-managed rows are read-only
Peers cursors, last success or error, received, rejections by reason, relayed trash
Tracking ads, postings, file size, overlay, last build, and a rebuild button
Config the effective config with secrets redacted, and a reload button (as SIGHUP)

Every change that affects trust (vote, pin, dup, origin policy) calls Library.Refresh in the admin’s root context, so the index follows even if the request goes away.

Formats, language, assets