09 · The leaderboard — what won, and why¶

Assumes you've skimmed 01–08. This notebook synthesizes them.

The whole point of the project: every technique implements one interface and is scored by one harness, so they're directly comparable on one board. Each technique notebook ran its own eval live; this notebook reads the canonical board that scripts/run_eval.py writes (the single source of truth — we don't re-derive 18 rows here) and tells the story across all of them.

In [1]:

Copied!

import pandas as pd

from courserec.config import RESULTS_DIR

pd.set_option("display.max_colwidth", 40)
KEY = ["name", "ndcg@10", "ndcg@10_ci_low", "ndcg@10_ci_high", "recall@10", "same_subject@10", "diversity", "query_latency_ms"]

board = pd.read_csv(RESULTS_DIR / "leaderboard.csv").sort_values("ndcg@10", ascending=False)
print(f"{len(board)} technique×config rows on the cross-listing board\n")
board[KEY].reset_index(drop=True)
import pandas as pd

from courserec.config import RESULTS_DIR

pd.set_option("display.max_colwidth", 40)
KEY = ["name", "ndcg@10", "ndcg@10_ci_low", "ndcg@10_ci_high", "recall@10", "same_subject@10", "diversity", "query_latency_ms"]

board = pd.read_csv(RESULTS_DIR / "leaderboard.csv").sort_values("ndcg@10", ascending=False)
print(f"{len(board)} technique×config rows on the cross-listing board\n")
board[KEY].reset_index(drop=True)

18 technique×config rows on the cross-listing board

Out[1]:

	name	ndcg@10	ndcg@10_ci_low	ndcg@10_ci_high	recall@10	same_subject@10	diversity	query_latency_ms
0	sbert(all_minilm_l6_v2,idx=flat)	0.9710	0.9645	0.9770	1.0000	0.2177	0.7272	0.326
1	sbert(all_mpnet_base_v2,idx=flat)	0.9706	0.9640	0.9767	1.0000	0.2308	0.7332	0.560
2	llm_rerank(qwen3_8b,base=sbert_all_m...	0.9649	0.9574	0.9718	0.9921	0.2146	0.7415	0.341
3	nmf(k=50,sw=on,ng=1-1,tw=1)	0.9604	0.9532	0.9683	0.9972	0.1066	0.8207	0.186
4	rerank(cross_encoder_ms_marco_minilm...	0.9604	0.9527	0.9673	1.0000	0.2083	0.7338	74.009
5	bm25(sw=on,ng=1-1,tw=3,k1=1.5,b=0.75)	0.9582	0.9505	0.9661	0.9972	0.1920	0.7395	3.001
6	tfidf(sw=on,ng=1-2,tw=1)	0.9580	0.9505	0.9658	0.9981	0.1918	0.6876	33.727
7	bm25(sw=on,ng=1-1,tw=1,k1=1.5,b=0.75)	0.9577	0.9502	0.9655	0.9972	0.1869	0.7387	2.982
8	llm_tags(qwen3_8b)	0.9569	0.9498	0.9651	0.9972	0.1804	0.7217	0.781
9	lsa(k=200,sw=on,ng=1-1,tw=1)	0.9566	0.9487	0.9642	0.9972	0.1860	0.7133	0.273
10	tfidf(sw=on,ng=1-1,tw=1)	0.9553	0.9480	0.9634	0.9972	0.1905	0.7230	3.025
11	tfidf(sw=on,ng=1-1,tw=3)	0.9527	0.9442	0.9608	0.9972	0.1896	0.7265	3.132
12	lda(k=50,sw=on,ng=1-1,tw=1)	0.9521	0.9439	0.9599	0.9972	0.0786	0.8307	0.184
13	metadata(text=0.9,facets=subject+dep...	0.9476	0.9396	0.9563	0.9981	0.2086	0.6754	0.931
14	rerank(cross_encoder_ms_marco_minilm...	0.9439	0.9350	0.9517	1.0000	0.1868	0.8227	75.187
15	metadata(text=0.7,facets=subject+dep...	0.9362	0.9274	0.9448	0.9981	0.4275	0.7074	0.924
16	metadata(text=0.5,facets=subject+dep...	0.9093	0.8995	0.9192	0.9981	0.7979	0.8258	0.932
17	rerank(cross_encoder_ms_marco_minilm...	0.8199	0.7999	0.8372	0.8006	0.1608	0.8939	73.811

1. The headline race, with error bars¶

NDCG@10 per technique with its bootstrap CI. The decisive fact isn't just the order — it's how much the CIs overlap. Techniques whose intervals overlap are statistically a tie, however different their point estimates look.

In [2]:

Copied!





import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt

from nbtools import plot_metric_ci

# de-clutter: shorten names and show the spread of distinct families
show = board.drop_duplicates("name").head(12)
labels = [n.split("(")[0] + ("…" if "(" in n else "") for n in show["name"]]
cis = list(zip(show["ndcg@10_ci_low"], show["ndcg@10_ci_high"]))
ax = plot_metric_ci(labels, show["ndcg@10"].tolist(), cis, title="Cross-listing lens · NDCG@10 (95% CI)")
plt.tight_layout()
ax.figure
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt

from nbtools import plot_metric_ci

# de-clutter: shorten names and show the spread of distinct families
show = board.drop_duplicates("name").head(12)
labels = [n.split("(")[0] + ("…" if "(" in n else "") for n in show["name"]]
cis = list(zip(show["ndcg@10_ci_low"], show["ndcg@10_ci_high"]))
ax = plot_metric_ci(labels, show["ndcg@10"].tolist(), cis, title="Cross-listing lens · NDCG@10 (95% CI)")
plt.tight_layout()
ax.figure

Out[2]:

<Figure size 700x700 with 1 Axes>

2. The cross-listing lens flatters near-duplicates¶

Cross-listed twins share near-identical text, so this lens is easy — even plain lexical methods cluster near the top, and the leaders sit inside each other's CIs. It validates correctness, not quality. Note the same_subject@10 column: high values are a sanity floor, not a goal (a subject-only model would max it while being useless).

In [3]:

Copied!





families = {
    "sbert": "semantic", "rerank": "semantic+rerank", "tfidf": "lexical",
    "bm25": "lexical", "lsa": "topic", "nmf": "topic", "lda": "topic",
    "metadata": "metadata", "llm_tags": "llm",
}
board["family"] = board["name"].str.split("(").str[0].map(families).fillna("other")
board.groupby("family")["ndcg@10"].agg(["max", "count"]).sort_values("max", ascending=False)
families = {
    "sbert": "semantic", "rerank": "semantic+rerank", "tfidf": "lexical",
    "bm25": "lexical", "lsa": "topic", "nmf": "topic", "lda": "topic",
    "metadata": "metadata", "llm_tags": "llm",
}
board["family"] = board["name"].str.split("(").str[0].map(families).fillna("other")
board.groupby("family")["ndcg@10"].agg(["max", "count"]).sort_values("max", ascending=False)

Out[3]:

	max	count
family
semantic	0.9710	2
other	0.9649	1
semantic+rerank	0.9604	3
topic	0.9604	3
lexical	0.9582	5
llm	0.9569	1
metadata	0.9476	3

3. The free-text lens — where meaning matters¶

The judged-query board scores recommend_by_text on hand-labeled queries — the only lens that measures free-text search, the mode semantic and topic methods are built to win (and lexical, blind to synonyms, is meant to struggle).

In [4]:

Copied!

text_board = pd.read_csv(RESULTS_DIR / "leaderboard_text.csv").sort_values("ndcg@10", ascending=False)
text_board[["name", "ndcg@10", "recall@10", "mrr"]].head(8).reset_index(drop=True)
text_board = pd.read_csv(RESULTS_DIR / "leaderboard_text.csv").sort_values("ndcg@10", ascending=False)
text_board[["name", "ndcg@10", "recall@10", "mrr"]].head(8).reset_index(drop=True)

Out[4]:

	name	ndcg@10	recall@10	mrr
0	sbert(all_minilm_l6_v2,idx=flat)	0.6821	0.7056	0.8083
1	llm_rerank(qwen3_8b,base=sbert_all_m...	0.6559	0.6671	0.7924
2	sbert(all_mpnet_base_v2,idx=flat)	0.6353	0.6308	0.8091
3	rerank(cross_encoder_ms_marco_minilm...	0.6098	0.6215	0.7718
4	rerank(cross_encoder_ms_marco_minilm...	0.5027	0.5078	0.7898
5	tfidf(sw=on,ng=1-1,tw=3)	0.4991	0.5514	0.6156
6	bm25(sw=on,ng=1-1,tw=3,k1=1.5,b=0.75)	0.4923	0.5226	0.6313
7	tfidf(sw=on,ng=1-2,tw=1)	0.4870	0.5335	0.6264

4. The graph's own board (not comparable)¶

The graph reads cross-listings, so it's scored only on held-out edges — a different task ("recover a removed twin") on a different board. It is never ranked against the content rungs above; comparing them would be apples to oranges.

In [5]:

Copied!

held = pd.read_csv(RESULTS_DIR / "leaderboard_heldout.csv").sort_values("ndcg@10", ascending=False)
held[["name", "ndcg@10", "recall@10", "n_queries"]].reset_index(drop=True)
held = pd.read_csv(RESULTS_DIR / "leaderboard_heldout.csv").sort_values("ndcg@10", ascending=False)
held[["name", "ndcg@10", "recall@10", "n_queries"]].reset_index(drop=True)

Out[5]:

	name	ndcg@10	recall@10	n_queries
0	sbert(all_minilm_l6_v2,idx=flat)	0.9126	1.0000	388
1	llm_rerank(qwen3_8b,base=sbert_all_m...	0.9126	1.0000	388
2	sbert(all_mpnet_base_v2,idx=flat)	0.9111	1.0000	388
3	rerank(cross_encoder_ms_marco_minilm...	0.9046	1.0000	388
4	tfidf(sw=on,ng=1-1,tw=1)	0.8945	0.9897	388
5	bm25(sw=on,ng=1-1,tw=3,k1=1.5,b=0.75)	0.8940	0.9897	388
6	tfidf(sw=on,ng=1-2,tw=1)	0.8929	0.9923	388
7	llm_tags(qwen3_8b)	0.8908	0.9897	388
8	nmf(k=50,sw=on,ng=1-1,tw=1)	0.8898	0.9897	388
9	tfidf(sw=on,ng=1-1,tw=3)	0.8870	0.9897	388
10	lda(k=50,sw=on,ng=1-1,tw=1)	0.8858	0.9897	388
11	rerank(cross_encoder_ms_marco_minilm...	0.8839	1.0000	388
12	metadata(text=0.9,facets=subject+dep...	0.8833	0.9923	388
13	bm25(sw=on,ng=1-1,tw=1,k1=1.5,b=0.75)	0.8825	0.9897	388
14	lsa(k=200,sw=on,ng=1-1,tw=1)	0.8718	0.9897	388
15	metadata(text=0.7,facets=subject+dep...	0.8618	0.9923	388
16	metadata(text=0.5,facets=subject+dep...	0.8292	0.9923	388
17	rerank(cross_encoder_ms_marco_minilm...	0.7437	0.7443	388
18	graph(meta=off,wx=1.0,wm=0.3,c=0.15)	0.1311	0.2294	388
19	graph(meta=on,wx=1.0,wm=0.3,c=0.15)	0.1300	0.2294	388

5. The story across all eight techniques¶

Rung	Notebook	Verdict on the cross-listing lens
Lexical (TF-IDF, BM25)	01	strong — twins share text; the correctness floor
Topics (LSA/NMF/LDA)	02	~ties lexical; pays off in interpretability + free text
Semantic (SBERT)	03	tops the board, and actually does free-text search
Rerank + MMR	04	marginal here (twins already on top); MMR buys diversity
Metadata fusion	05	hurts — structure pulls non-twins up (honest negative)
Graph (PPR)	06	own held-out board; not comparable
Clustering	07	not a ranker — a diagnostic
LLM	08	lost at ranking, won at explaining

6. Takeaways¶

SBERT wins the primary lens and free text — the one rung strong at both, which is why it's the UI default.
The cross-listing race is tight and CI-bound: never crown a winner on a sub-CI gap. The interesting results are the negatives — metadata fusion hurting, the LLM rungs losing — which a single-lens reading would hide.
Different goals need different boards: free-text (judged queries) and graph (held-out edges) are separate tasks, scored separately. One number is never the whole story — that's the entire reason the harness reports three lenses.

Source: scripts/run_eval.py, courserec/eval.py · regenerate the boards with python scripts/run_eval.py. You've reached the end of the path — back to the index.