ADR-0003: A hand-labeled judged-query set for the free-text lens

Date: 2026-06-03 Status: Accepted

Context

Through Phase 2 the leaderboard measured only item-to-item similarity on the cross-listing lens. That lens is near-trivial for lexical methods (twins share text) and could not separate any technique — every CI overlapped. But the whole premise of topic and semantic methods is that they win on free-text queries ("practical deep learning"), via recommend_by_text — a mode with no automatic ground truth (plan §3 lens 3, §6.3). Without a way to score it, adding more powerful techniques means shipping them into a harness that cannot tell whether they are actually better. Closing this gap was chosen ahead of, then alongside, Phase 3.

Two design points were non-obvious:

1. Where the ground truth comes from. There are no clicks or ratings, and the relevant set for a query is open-ended (any of 11k courses could be on-topic).
2. How not to bias the labels toward the methods under test. If labels were generated by running a lexical recommender and keeping its top hits, the lens would reward lexical methods by construction.

Decision

1. Hand-label a small natural-query set. data/judged_queries.json holds 22 natural-language queries, each with 3–9 course_ids a human judged clearly on-topic (125 labels total, spanning 34 subjects). Queries are deliberately phrased in words that differ from course titles (e.g. "building web applications" for courses titled "Front-End Web Architecture") so the lens rewards meaning, not surface overlap.
2. Treat the set as relative, not absolute, ground truth. It is curated and necessarily incomplete, so absence from a relevant set is not evidence of irrelevance. Recall is read as a comparison across techniques on the same set, never as an absolute hit rate.
3. Score with the same metrics as the primary lens. score_text_queries (eval.py) reuses the binary ranking metrics and the NDCG@10 bootstrap CI via the shared _aggregate_ranking_metrics helper. same_subject@10 is undefined for a query with no seed subject and is reported as NaN (dropped from the text leaderboard).
4. Version-control the labels; commit them. The set is curated ground truth, so it is force-committed via a .gitignore exception (!data/judged_queries.json) even though the rest of data/ is ignored. scripts/build_judged_queries.py validates every label resolves against the catalog (--validate, exit-coded for CI), summarizes the set (--stats), and can suggest lexical candidates to seed — never decide — hand labels (--suggest).
5. Skip + flag, never silently omit. Item-to-item-only techniques that raise NotImplementedError from recommend_by_text are detected (recommender_supports_text) and listed as skipped in the text leaderboard.

Alternatives considered

Option	Pros	Cons	Why rejected
LLM-as-judge to rate top-k relevance	Scales to many queries; no manual labeling	Needs a key (breaks local-only); must itself be validated against human labels before trusting	Deferred to Phase 7; a hand set is the validation baseline it would need anyway
Derive relevant sets from a lexical recommender's output	Fully automatic	Biases ground truth toward lexical methods — defeats the lens	A lens that rewards the method under test measures nothing
Use department/subject as the relevant set for a topic query	Automatic, large	Subject ≠ topic; rewards the useless same-subject-only model (plan §3 lens 2)	Already covered (and cautioned against) by the same-subject floor
Store the set under data/ (gitignored)	Consistent with other data	Ground truth would not be committed; eval not reproducible on a fresh clone	Curated labels must travel with the repo

Consequences

Positive: Free-text mode is finally measurable, and the lens discriminates where cross-listing could not — on this set NDCG@10 spreads from ~0.61 (lexical / SBERT) down to ~0.07 (NMF/LDA at k=50), so it cleanly separates methods that collapse on short queries from those that don't. Phase 3 lands in a harness that can score it. Negative: 22 queries / 125 labels is small, so text-lens CIs are wide (~±0.11 on NDCG@10) — differences among the top methods are not significant. Labels reflect one labeler's judgment and a single catalog snapshot. Neutral: Maintenance cost — a future catalog revision can strand a label; load_judged_queries drops stale ids with a warning and the validator catches them, so the set degrades loudly, not silently.

Implementation notes

src/courserec/eval.py: JudgedQuery, load_judged_queries, score_text_queries, recommender_supports_text, and the extracted _aggregate_ranking_metrics (shared with score_crosslist). scripts/build_judged_queries.py: validate / stats / suggest. scripts/run_eval.py: writes a companion results/leaderboard_text.{md,csv}. Tests in tests/test_eval.py cover stale-id dropping, the metrics, the NaN subject, and the capability probe. See also ADR-0004.