Multi-method IR evaluation: 6 retrieval methods, 768 CS faculty from 9 US universities, 162 graded relevance judgments across 5 queries. Reranked fusion achieves best mean NDCG@10 (0.477) across queries; TF-IDF is significantly worse than all other methods (p < 0.001, Bonferroni corrected).
| # | Name | University | Research | Semantic | BM25 | TF-IDF | Hybrid | Reranked |
|---|
UMAP projection of 384-dimensional sentence embeddings. Each dot is a professor, colored by research cluster. The ★ star marks your research interest position.
5-query evaluation (162 graded relevance judgments):
Reranked fusion (BM25 + TF-IDF + Semantic) achieves the best mean NDCG@10 across all 5 queries (0.477), followed by Semantic (0.450), Hybrid (0.421), BM25 (0.406), Jaccard (0.303), and TF-IDF (0.246).
After Bonferroni correction across 15 method pairs, TF-IDF is significantly worse than all other methods (p < 0.001). No other pair is distinguishable at 5 queries.
Single-query Q1 evaluation (67 labels):
TF-IDF and Semantic top-10 agree on only 40% of professors (Spearman ρ = 0.624). Semantic achieves 50% Precision@10 vs 10% for TF-IDF — a 5× improvement on Q1.
Field ablation:
Biography is the most important profile field — removing it drops NDCG@10 by 0.130 (22%). Research area tags alone score 0.427.
arXiv concatenation (controlled experiment):
Appending paper abstracts reduces NDCG@10 by 0.176 on the same subpopulation (0.772 → 0.596). Relevant professors are hurt more than irrelevant ones (avg Δ −0.238 vs −0.139), supporting jargon-dilution as the mechanism. A late-fusion architecture is the recommended fix.
GradientBoosting model trained on 67 relevance labels. NDCG@10 = 0.568
Impact of removing each data component on NDCG@10