Results

Findings as each phase lands. The leaderboard itself (results/leaderboard.md) is regenerated by python scripts/run_eval.py and never hand-edited; this file is the interpretation.

The "why this fits" explainer — closing out the LLM enrichment track

The last LLM-enrichment piece (RecommendationExplainer, recommenders/llm.py) is the one the two negative results pointed to. The tag technique (ADR-0009) and the reranker (ADR-0010) both tried to make the LLM rank, and both lost to a 0.38 M-param SBERT bi-encoder already sitting at the recall ceiling. Explanation is a different job: the ranking already exists (from SBERT), and the LLM is spent only on justifying one (query, candidate) pair — a generative task with no ground-truth ordering to beat.

So, by design, this is not a recommender and not on the leaderboard. It returns no list[Rec], is never scored by eval.py, and has no metric row — scoring a free-text justification against a ranking ground truth would be a category error. Quality is assessed by inspection. One deterministic Ollama call (qwen3:8b) per pair returns a single sentence under a {"reason": str} schema, cached by sha1(model + query + candidate-id) in explanations.json. Because the "why" line is optional in the UI, every unavailability path (empty query, blank reason, Ollama down with no cache) degrades to None — the UI simply omits the line; only an unknown candidate id raises. Zero new deps (reuses the ADR-0009 client).

Validated live (2026-06-20, qwen3:8b) via scripts/explain_recs.py over SBERT recommendations, both modes:

• COMPSCI 189 → STAT C241A — "Both courses cover statistical learning theory, classification, regression, clustering, dimensionality reduction, and ensemble methods."
• query "ethics of artificial intelligence" → PHILOS 14 — "…explores ethical issues in AI, such as algorithmic bias and moral responsibility, aligning with the query's focus…"

Concrete and on-topic; the cache persists and a repeat pair serves with no call. Honest caveats: no automatic quality metric (inspection only — an LLM-as-judge is a possible future check, itself needing validation); qwen3:8b-specific output that occasionally restates the query rather than naming the shared concept; the explainer describes a recommendation, it does not certify it. The LLM enrichment track is complete: tags → reranker → explainer, two negative ranking results and one shipped UI helper. See ADR-0011.

The zero-shot LLM reranker (measured: does not beat the base)

The reranker (LLMRerankRecommender, recommenders/llm.py) acts on the tag technique's negative lesson: keep a strong retriever and spend the LLM on the full candidate text, not on lossy distillation. It retrieves the top-20 from the MiniLM SBERT base, then reorders those candidates with one deterministic Ollama call (qwen3:8b) that returns an integer permutation under a JSON-schema format; reranks cache by sha1(model+query+candidate-ids), and it falls back to the base order when Ollama is down. Design in ADR-0010.

Headline — a second documented negative result (qwen3:8b)

Measured 2026-06-19 on a warm-cache run_eval (1072 cross-listing seeds, 44 judged queries). The reranker does not beat the SBERT MiniLM base on either lens:

Technique	Cross-list NDCG@10	95% CI	Free-text NDCG@10	95% CI
sbert (MiniLM) — base	0.9710	[0.965, 0.977]	0.6821	[0.615, 0.746]
llm_rerank (qwen3:8b)	0.9649	[0.957, 0.972]	0.6559	[0.586, 0.729]
Δ vs base	−0.006	(CIs overlap)	−0.026	(CIs overlap)

• Both deltas are negative and inside the bootstrap CIs — no significant difference. The reranker also lowers recall@10 (cross-list 1.000 → 0.992, text 0.706 → 0.667): it occasionally reorders a true twin out of the top-10.
• NDCG@5 on the text lens is the only flat bright spot (0.6420 → 0.6457, +0.004, also within noise) — the reranker tidies the very top but not @10.
• Why: recall@20 is identical for base and reranker (1.000 cross-list, 0.869 text), confirming a pure reorder. SBERT's top-20 is already at/near the recall ceiling, so there is essentially nothing to fix and only room to hurt — the same "twins already rank first" trap that sank the rerank stage's cross-encoder, now with a zero-shot LLM standing in for the trained one.
• Cost: ~4.4 s/query (cross-list) and ~3.7 s/query (text) on a cold cache — ~13000× the base's sub-millisecond latency. The warm cache reruns offline and is reproducible (metric columns byte-identical across runs).

Verdict: SBERT MiniLM stays the top ranking-lens technique. Like the tag technique, the LLM reranker is honest portfolio evidence — the mechanism is sound and robust, but a strong retriever on a near-ceiling task leaves it no headroom. A weaker base (TF-IDF) or a larger model (qwen3:32b) are the obvious follow-ups; neither was run.

LLM enrichment via local Ollama (the tag-extraction technique)

The LLM stage adds the LLM technique (recommenders/llm.py): a local LLM (Ollama, qwen3:8b, no API key — see the project plan) reads each course and emits structured tags (topics, skills, level, prereqs-mentioned); the technique ranks by TF-IDF cosine over those tag profiles (topics+skills+prereqs), with raw-text fallback for any course not yet enriched. Enrichment is a separate, cached, resumable pass (scripts/enrich_catalog.py) so fit only reads the cache and the eval stays fast. See ADR-0009.

Headline — full enrichment overturns it: distillation is not competitive

The catalog is now 100% enriched (10,900/11,073 courses carry non-empty tags; the rest are genuinely text-free). That removes the target/distractor vocabulary-separation confound — and the verdict flips.

Technique	Cross-list NDCG@10	95% CI	Free-text NDCG@10	95% CI
sbert (MiniLM)	0.9710	[0.965, 0.976]	0.6821	[0.613, 0.747]
best bm25 / tfidf-bigram	~0.958	[0.951, 0.966]	~0.49	[0.41, 0.57]
llm_tags (qwen3:8b) — full	0.9569	[0.950, 0.965]	0.4041	[0.327, 0.468]
tfidf (unigram, baseline)	0.9553	[0.947, 0.964]	0.4607	[0.388, 0.524]
llm_tags — partial (12.5%), for contrast	0.9603	[0.952, 0.967]	0.5847	[0.503, 0.662]

• Cross-listing: llm_tags falls to 0.957 — statistically tied with the lexical cluster (its CI overlaps tfidf/bm25), mid-pack. Distillation neither helps nor hurts twin recovery: twins share near-identical text, so they share near-identical tags, exactly like lexical.
• Free-text: llm_tags drops to 0.404 — below plain TF-IDF (0.461), recall 0.55 → 0.38. The provisional "+0.12 over lexical" was almost entirely the partial-enrichment artifact; with distractors in the same tag space the apparent win evaporates and reverses.

Why distill-to-tags loses (the honest mechanism)

Compressing a rich 50–200-term description into ~6–12 abstract tags throws away discriminative detail that raw TF-IDF exploits, and the loss outweighs the synonym-normalization the LLM adds:

1. The tag vocabulary saturates at catalog scale. Across 11k courses many share generic tags ("machine learning", "data analysis", "research methods"), so the profiles stop discriminating — fine for separating a handful of targets from raw-text distractors (the partial run), useless for ranking gold among 11k tag-profiles (the full run).
2. Query/corpus tag surface forms don't always align. A free-text query is enriched to its own tags, which need not match the course tags lexically ("deep learning" query vs "neural networks" course tag), so tag-to-tag overlap is less reliable than raw term overlap on this paraphrase-extreme judged set.
3. Extraction quality was never the problem. Spot checks are clean (AEROENG C124 → composite materials, mechanical properties, aircraft structures, nanocomposites). The features are sound; the architecture — lossy distillation then lexical matching — is what doesn't pay off.

What this says / next step

• Honest verdict: the tag-distillation technique is not competitive. It ties lexical on cross-listing and underperforms plain TF-IDF on free text. Running the full catalog (enrich_catalog.py --all) was the point — it converted a promising-but-confounded number into a clean negative, which is the result that belongs on the leaderboard (boards now carry an "LLM enrichment (full)" note, no longer the partial caveat). A textbook case for why you de-confound before believing a win.
• The LLM's value, if any, is elsewhere. Not in compressing text to tags and matching lexically, but in operations that use the full candidate text: the zero-shot reranker (judge candidate pairs directly) and the "why this fits" explanation, plus LLM-as-judge validation of the free-text lens (part of the evaluation methodology). Those are the next builds.
• Still local, keyless, zero-cost, zero new dependencies. The technique runs on Ollama over stdlib urllib; the offline guarantee is untouched, and it skips+flags (never hard-fails) when Ollama is down and nothing is cached. The 100% tag cache is reusable by the reranker and the UI regardless of this technique's ranking verdict.

Clustering + 2-D map over the SBERT embeddings (diagnostic)

The clustering stage adds src/courserec/cluster.py and scripts/run_clustering.py — a diagnostic, not a ranker. It does not subclass Recommender or join any leaderboard; it asks whether the SBERT semantic space has structure. Output: results/cluster_report.md and a subject-colored 2-D map at results/plots/embedding_map.png. See ADR-0007.

Headline — the embedding space is a smooth manifold, not tidy clusters

Three clusterings of the 11,073 MiniLM vectors (all-MiniLM-L6-v2):

algorithm	n_clusters	n_noise	silhouette	subject_purity	largest_frac
kmeans (k=100)	100	0	0.116	0.328	0.024
agglomerative (ward, k=100)	100	0	0.082	0.324	0.026
hdbscan (min=15)	32	9,955	0.636	0.282	0.015

What this says

1. Low silhouette under forced partitions. KMeans/agglomerative score ~0.08–0.12 — points sit only slightly closer to their own centroid than the next. The vectors form a continuous topic manifold, not isolated blobs.
2. HDBSCAN confirms it from the other side. Asked to find density, it labels 90% of the catalog as noise and keeps only 32 dense cores; its high 0.64 silhouette is survivorship — it scores only the easy points it kept.
3. Subjects are recovered ~32% from text alone. No metadata is used, yet a cluster's plurality subject covers ~1/3 of its members (well above the ~1/242-subject floor) — coherent neighborhoods exist, they just blend at the edges. The 2-D map shows exactly this: MECENG, MUSIC, ANTHRO etc. form visible local regions inside one connected cloud.
4. No degenerate giant cluster — the largest is ~2.5% of the catalog, so the moderate purity is real coherence, not one blob swallowing everything.

Why this matters for the rankers

The smoothness is why semantic similarity works for free-text and related-but- not-twin queries (the SBERT semantic-vector stage) yet barely beats lexical on near-duplicate twins: in a continuous space, "close" is graded, not categorical. It also frames the coverage/diversity numbers — with no hard cluster walls, top-k lists can wander a neighborhood rather than collapse onto a clique.

Honest limitations

• Projector fell back to t-SNE. umap-learn is in the optional viz extra but not installed here, so the map is t-SNE (also a valid projection choice); UMAP would give a faster, more global-structure-faithful layout.
• Cluster count is a chosen knob. k=100 is a coarse default (242 subjects); the coherence numbers shift with k. This is a shape probe, not a tuned model.
• Silhouette is sampled (2,000 points) for tractability — a point estimate, not a CI.

Metadata fusion (weighted one-hot facets ⊕ TF-IDF)

The metadata-fusion stage's ranker (recommenders/metadata.py) fuses a TF-IDF text block with a one-hot subject + department + level + units block under a single weight λ (text_weight), L2-normalizing each block so the fused score is λ²·cos_text + (1−λ)²·cos_meta. λ=1.0 is bit-for-bit the tfidf baseline, so any delta is the metadata's doing. It joins the cross-listing + judged-text leaderboards; see ADR-0008.

Headline — adding metadata hurts the primary lens, monotonically

Technique	Cross-list NDCG@10	95% CI	text-lens NDCG@10
tfidf (unigram, baseline)	0.9553	[0.947, 0.964]	0.4607
metadata (λ=0.9)	0.9476	[0.939, 0.956]	0.4607
metadata (λ=0.7)	0.9362	[0.928, 0.944]	0.4607
metadata (λ=0.5)	0.9093	[0.899, 0.919]	0.4607

Cross-listing NDCG@10 falls the moment metadata enters and keeps falling as λ drops — every fused config sits below the pure-text baseline. On the judged free-text lens all three λ tie the baseline exactly (0.4607): a query carries no facets, so its metadata block is zero and ranking collapses to pure TF-IDF, by design.

Why it's a clean mechanistic loss, not a bug

99.7% of cross-listing edges connect different subjects, and 97.0% different departments — a cross-listing is the same course offered under two subject codes. So the one-hot subject/department block, which raises similarity for same-subject courses, actively pushes a twin away from its seed. Fusion can only hurt the one lens that rewards twins ranking each other. This is the same tension the graph technique hit from the structural side (the graph stage: "cross-listed twins frequently span subjects"); metadata fusion quantifies it from the content side.

What this says

• These facets are adversarial to the cross-listing target, not merely orthogonal. Their genuine value is browse/filter coherence — "more grad Mechanical Engineering courses like this one" — which the harness only measures as the weak same-subject proxy it explicitly warns against optimizing (rules/eval.md). With no browse lens in the plan's three, the leaderboard can only show the cost, so the cost is what's documented.
• One real win survives: sparse-text robustness. A description-less course has a near-empty text block but a fully-populated metadata block, so fusion still places it sensibly where text-only methods have nothing to go on — it just doesn't show up in a twin-recovery metric.
• Self-contained and legible. TF-IDF backend → fully sparse, no extra/key, and the λ=1 ≡ baseline identity makes the ablation exact. λ is swept, not optimized; the gradient is monotone toward pure text, so finer resolution wouldn't change the direction.

The course graph — PPR on a held-out cross-listing edge split

The graph stage adds the graph technique (recommenders/graph.py) — the one technique allowed to read Cross-Listed Course(s) — and the machinery that keeps that read leakage-free: a 30% held-out edge split (eval.split_crosslist_edges) and a third leaderboard, results/leaderboard_heldout.md, where the graph and every content method predict the same 219 withheld edges (388 seeds). The graph ranks by personalized-PageRank proximity (random walk with restart) over a course graph with subject/department auxiliary nodes; two configs (meta=on/meta=off) isolate the metadata glue. See ADR-0006.

Headline — on held-out twins, text crushes structure

Technique	Held-out NDCG@10	95% CI	Recall@10	same-subj@10	diversity
sbert (MiniLM)	0.9126	[0.895, 0.928]	1.000	0.205	0.692
tfidf (unigram)	0.8945	[0.873, 0.914]	0.990	0.186	0.232
nmf (k=50)	0.8898	[0.869, 0.909]	0.990	0.097	0.240
graph (meta=off)	0.1311	[0.109, 0.155]	0.229	0.000	0.014
graph (meta=on)	0.1300	[0.108, 0.155]	0.229	0.816	0.867

The content methods score ~0.89–0.91 — essentially their full-truth numbers — because a held-out edge costs a text method nothing: cross-listed twins share near-identical title+description, so the twin is still rank 1 from text alone. The graph, the only method actually doing leakage-free link prediction, recovers just ~23% of held-out twins (NDCG@10 0.131, CI [0.109, 0.155] — far below every content row).

Why the graph hits a ceiling — and why metadata can't lift it

1. Most cross-listings are isolated pairs. Mean ~1.35 twins/seed: when a pair's only edge is the one held out, no walk can reach the twin — there is no surviving path. The ~23% the graph does recover are edges inside larger (3+-way) cross-listing components, where transitivity survives one removal. This is the structural ceiling, and the held-out split exists precisely to expose it rather than let the graph "win" by reading its own target.
2. Metadata glue raises coverage, not recovery. Turning subject/department aux nodes on barely moves NDCG@10 (0.130 vs 0.131 — a tie inside the CI) yet transforms the list shape: same-subject@10 jumps 0.00 → 0.82, diversity 0.01 → 0.87, coverage 0.014 → 0.182. The walk now floods the top-k with same-subject/department neighbors — but cross-listed twins frequently span subjects (that is what makes them interesting), so those neighbors are almost never the withheld twin. The glue makes recommendations look fuller without recovering the target.

What this says

• The graph is not a similarity ranker for this catalog, and the eval proves it honestly. Where text is near-identical (twins), structure adds nothing a bag-of-words model didn't already have. A graph earns its keep when edges encode signal absent from text — prerequisite chains, co-enrollment, curricular sequence — none of which this catalog has. Documented, not hidden.
• The held-out discipline worked. The graph reads the leak column yet is scored only on edges withheld from training; the comparison sits in its own leaderboard with a header note that it is a harder task than the full-truth file, so nothing is crowned across incomparable numbers.
• PPR cost is trivial (~0.4 ms/query meta=off, ~2.6 ms meta=on; fit < 0.05 s) and needs zero new dependencies — node2vec was rejected to avoid a heavy gensim dependency for a method whose ceiling is structural, not encoder-bound.

Retrieve → cross-encoder rerank → MMR (+ a harder judged set)

Two things landed together: the rerank technique (a MiniLM bi-encoder retrieves the top 50, the ms-marco-MiniLM-L-6-v2 cross-encoder rescores, MMR re-orders with a λ knob) and a 2× expansion of the judged set (22 → 44 paraphrase-extreme queries, 309 labels over 80 subjects). The leaderboard is now 13 rows per lens.

Growing the judged set made the semantic win significant

The old 22-query set left SBERT's free-text lead inside the CI. On the larger, harder, deliberately paraphrase-extreme set the gap is now decisive:

Technique (config)	Free-text NDCG@10	95% CI	Recall@10
sbert (MiniLM)	0.6821	[0.6153, 0.7456]	0.706
sbert (MPNet)	0.6353	[0.5619, 0.7114]	0.631
rerank (λ=1.0)	0.6098	[0.5303, 0.6890]	0.622
tfidf (unigram, tw=3)	0.4991	[0.4120, 0.5847]	0.551
bm25 (tw=3)	0.4923	[0.4054, 0.5757]	0.523

SBERT MiniLM's CI [0.615, 0.746] clears the best lexical config's [0.412, 0.585] with no overlap — the semantic advantage the lens was built to detect is now real, not within-noise. Two levers did it: more queries (tighter CIs) and harder phrasing (queries whose words deliberately differ from the relevant courses' titles, which is exactly where lexical falls and meaning wins).

The MMR knob works — diversity moves with λ

The rerank stage's acceptance criterion is that the intra-list diversity metric moves with λ. It does, monotonically, on both lenses:

λ	Cross-listing NDCG@10	Cross-listing diversity	Free-text NDCG@10	Free-text diversity
1.0 (pure relevance)	0.9604	0.734	0.6098	0.745
0.5	0.9439	0.823	0.5027	0.822
0.3 (diversity-leaning)	0.8199	0.894	0.3857	0.870

Diversity is measured in the technique-agnostic TF-IDF reference space (ADR-0002), not the model's own — so the move is real, not self-flattering. Lower λ trades relevance for novelty exactly as designed.

Honest finding — the cross-encoder does not beat the bi-encoder here

This is the headline negative result. At λ=1.0 (pure rerank, no diversity penalty) the cross-encoder trails plain SBERT MiniLM on both lenses: free-text 0.610 vs 0.682, cross-listing 0.960 vs 0.971 — and costs ~70–80 ms/query versus sub-ms for the bi-encoder. Two reasons:

1. Domain mismatch. ms-marco-MiniLM-L-6-v2 is trained on web-search query→passage relevance; course-catalog text (and the twin-matching task in particular) is out of its training distribution.
2. Retrieval already wins. The bi-encoder places the cross-listed twin at rank 1; reranking only the top 50 can demote it but rarely promote a better one, so the cross-encoder mostly loses NDCG it cannot regain.

The value the rerank stage delivers is therefore the diversity control, not a relevance gain. A domain-tuned or fine-tuned cross-encoder might reverse this — out of scope for this phase, and a candidate next lever (see HANDOFF).

Semantic vectors (SBERT) + the judged free-text lens

Two things landed together: the judged free-text lens (the first scoring of recommend_by_text, the free-text evaluation lens) and the semantic technique (local SBERT, plus an API backend that skips with no key). The leaderboard is now 10 rows across two files — leaderboard.md (cross-listing) and leaderboard_text.md (judged queries, 22 hand-labeled).

The new lens earns its keep

The cross-listing lens still can't separate anyone (all CIs overlap). The free-text lens does — the same 10 techniques now spread across a wide range:

Technique (config)	Free-text NDCG@10	95% CI	Recall@10
sbert (MiniLM)	0.6169	[0.4978, 0.7261]	0.681
tfidf (unigram, tw=3)	0.6112	[0.4972, 0.7252]	0.736
sbert (MPNet)	0.6054	[0.4695, 0.7319]	0.635
bm25 (tw=3)	0.5840	[0.4650, 0.7051]	0.686
…
lsa (k=200)	0.3205	[0.4158]*	0.413
lda (k=50)	0.0744	—	0.092
nmf (k=50)	0.0684	—	0.067

What this says

1. The lens discriminates where cross-listing couldn't. Free-text NDCG@10 runs from ~0.62 down to ~0.07 — a real spread, not the tied ~0.95–0.97 huddle of the cross-listing lens. The thing that separates methods is exactly the thing the plan predicted: handling short queries whose words differ from course titles.
2. Topic models collapse on free text. NMF and LDA at k=50 score ~0.07 — near useless on recommend_by_text, despite topping/midding the cross-listing lens. Projecting a 2–4 word query through 50 topics leaves almost no signal; LSA at 200 components survives better (0.32) but is still far behind. This is the honest counterweight to the topic-model stage's interpretability story: those models are for reading the catalog, not for free-text retrieval.
3. Semantic leads — but not decisively. SBERT MiniLM has the top point estimate on both lenses (xlist 0.971, text 0.617) and the only perfect cross-listing Recall@10 (1.000). Yet on free text its lead over the best TF-IDF config (0.611) is well within the CI — a ~0.006 gap on a 22-query set. We do not crown it (the gap is within the CI).
4. Bigger ≠ better here. MPNet (768-d, ~170 s fit) did not beat MiniLM (384-d, ~9 s fit) on either lens. MiniLM is the better speed/quality trade; the larger model isn't justified by these numbers.
5. Queries are the fastest yet. Exact FAISS inner-product over normalized vectors answers in ~0.3 ms (MiniLM) — embeddings move the cost to a one-time encode, cached thereafter.

Honest limitations

• The judged set is small (22 queries, 125 labels). Text-lens CIs are wide (~±0.11), so no top-method difference is significant. The set is also one labeler's judgment on one catalog snapshot, and deliberately not paraphrase- extreme — which is likely why semantic doesn't pull away from lexical. A larger, harder query set is the clearest next lever. (Resolved in the rerank stage: the set grew to 44 paraphrase-extreme queries and the semantic lead is now significant — see the rerank stage above.)
• The API embedding row is unmeasured — skipped + flagged (no OPENAI_API_KEY; the repo runs local-only). Its graceful-skip path is tested; its live path is not.
• k is unswept for both topic and semantic — single configs, not tuned ceilings.

* Bootstrap CI columns abbreviated in this digest; see leaderboard_text.csv for the full table.

Topic models (LSA, NMF, LDA)

Same cross-listing lens (1,072 seeds), same primary metric. Topic models drop into the existing harness unchanged; the leaderboard is now 8 rows.

Technique (config)	NDCG@10	95% CI	Same-subj@10	Diversity	Query latency
nmf (k=50)	0.9604	[0.9529, 0.9685]	0.107	0.821	~0.20 ms
lsa (k=200)	0.9566	[0.9489, 0.9642]	0.186	0.713	~0.28 ms
lda (k=50)	0.9521	[0.9436, 0.9596]	0.079	0.831	~0.19 ms

What this says

1. Still no significant winner — and topic models don't beat lexical. NMF has the highest point estimate of all 8 techniques (0.9604), but its CI overlaps every lexical config and both other topic models. As the methodology predicts, the cross-listing lens can't reward topic models' real strength: twins already share near-identical text, so projecting to k topics can only blur a signal lexical methods already nail. This lens validates that the topic recommenders are correct, not that they're better.
2. They are the fastest queriers so far. A dense k-topic mat-vec answers in ~0.2 ms — ~15× faster than the lexical sparse mat-vec (~3 ms) — at the cost of a heavier fit (SVD/NMF/LDA factorization; LDA slowest).
3. They generalise past exact vocabulary. NMF and LDA pull far fewer same-subject neighbours into the top-10 (0.08–0.11 vs lexical's ~0.19) and return more diverse lists (diversity 0.71–0.83). The learned topics are coherent on inspection (e.g. an LDA topic ≈ ai, security, privacy, intelligence; an NMF topic ≈ history, political, economic, modern, asia).
4. LDA trails. Lowest NDCG and weakest same-subject coherence of the three — expected at only 50 topics over an 11k catalog with short course descriptions; the variational fit is also the slowest. It earns its place for interpretability and per-document topic distributions, not ranking quality.

Honest limitations

• The free-text gap is now the binding constraint. Topic and (coming) semantic methods are supposed to win precisely on synonym/paraphrase queries — exactly what recommend_by_text does and what the still-missing judged-query set (the free-text lens) would measure. Until that lens exists, every NDCG number here describes item-to-item similarity on near-duplicate text only.
• Topic count is unswept. k was fixed (LSA 200, NMF/LDA 50), not tuned; the point estimates above shouldn't be read as each method's ceiling.

Lexical baselines (TF-IDF, BM25)

Evaluated on the cross-listing lens (1,072 in-catalog seeds) with a same-subject sanity floor. Primary metric is NDCG@10 with a 95% percentile bootstrap CI.

Technique (config)	NDCG@10	95% CI	Recall@10	Query latency
bm25 (tw=3)	0.9582	[0.9505, 0.9661]	0.9972	~2.9 ms
tfidf (1–2 grams)	0.9580	[0.9505, 0.9658]	0.9981	~32 ms
bm25 (tw=1)	0.9577	[0.9502, 0.9655]	0.9972	~2.9 ms
tfidf (unigram)	0.9553	[0.9480, 0.9634]	0.9972	~2.9 ms
tfidf (unigram, tw=3)	0.9527	[0.9442, 0.9608]	0.9972	~3.0 ms

What this says

1. No statistically significant winner. All five configs sit at NDCG@10 ≈ 0.95–0.96 with fully overlapping CIs. Per the methodology we do not crown a winner on a sub-CI gap — the ranking by point estimate is noise.
2. The lens is near-trivial for lexical methods. Cross-listed twins share almost identical title+description text, so any bag-of-words model retrieves the twin at rank 1 (Recall@10 ≈ 0.997). This validates correctness of the harness and the recommenders far more than it discriminates quality — the reason the plan mandates additional lenses.
3. Latency is the real differentiator so far. BM25 and unigram TF-IDF answer in ~3 ms; the bigram TF-IDF vocabulary blows up to ~32 ms/query for no NDCG gain. Title-weighting (tw=3) barely moves any metric.
4. Same-subject coherence ≈ 0.19, i.e. ~2 of the top 10 share the seed's subject. Reported as a floor only — cross-listings frequently span subjects (that is what makes them interesting), so a low value here is fine.

Honest limitations

• Free-text mode is unscored. recommend_by_text runs, but there is no ground truth for it yet (needs the judged-query set — the free-text lens). Until then, NDCG numbers describe item-to-item similarity only.
• Coverage ≈ 0.43. Across all 1,072 seeds' top-10, lexical methods surface ~43% of the catalog — a popularity/redundancy skew worth watching as richer techniques arrive.
• Cross-listings cover only ~10% of courses and reward near-duplicate text; diversity/coverage/novelty are reported precisely because the primary metric can be "won" while recommending nothing useful.