sahel-sh/DeepHone

evaluate-and-etc

Evaluate a DeepHone deep-research run with the gpt-oss-120b LLM-as-judge (accuracy / recall / calibration), aggregate token usage, and compute the Effective Token Cost (ETC) plus the paper's accuracy-vs-cost figures (Figures 1-3). Use after producing run…

원문 언어: 영어

extend-searcher-reranker

Add a custom retriever (searcher) or reranker to DeepHone by implementing the BaseSearcher / BaseReranker interface and registering it in the SearcherType / RerankerType enum. Use when integrating a new retrieval or reranking method into the benchmark.

원문 언어: 영어

run-reranking-experiment

Run the paper's core experiments — one-shot reranking effectiveness (Table 1) and end-to-end deep-research with listwise or cross-encoder reranking at depth d in {10,20,50} and search reasoning in {low,medium,high} (Tables 2/4, Figures 2/3). Use to produce…

원문 언어: 영어

serve-vllm-models

Launch the vLLM servers needed for DeepHone experiments — the gpt-oss search agent (with tool calling), the reranker (gpt-oss listwise or Qwen3-Reranker-0.6B cross-encoder), and the gpt-oss-120b LLM-as-judge. Use before running or evaluating deep-research…

원문 언어: 영어

setup-benchmark

Set up the DeepHone / BrowseComp-Plus benchmark — install the uv environment (Python 3.10, Java 21, flash-attn), decrypt the dataset, and download or build the BM25 / Qwen3-Embedding-8B retrieval indexes. Use this before running any experiment or evaluation.

원문 언어: 영어