run-reranking-experiment

name	run-reranking-experiment
description	Run the paper's core experiments — one-shot reranking effectiveness (Table 1) and end-to-end deep-research with listwise or cross-encoder reranking at depth d in {10,20,50} and search reasoning in {low,medium,high} (Tables 2/4, Figures 2/3). Use to produce run directories over BrowseComp-Plus.

Run a reranking experiment

Requires the indexes (setup-benchmark) and running model servers (serve-vllm-models). --first-stage-k is the reranking depth d; candidates are truncated to 512 tokens; the agent sees top---k 5.

One-shot reranking effectiveness (Table 1 / Figure 1)

Retrieve top-d, rerank, write a TREC run, then score it:

python scripts_retrieval_only/retrieve.py \
    --searcher-type faiss --index-path "indexes/qwen3-embedding-8b/corpus.shard*.pkl" \
    --model-name Qwen/Qwen3-Embedding-8B --normalize \
    --query topics-qrels/queries.tsv --output-dir retrieval_output --k 5 \
    --reranker-type batch_listwise_vllm --reranker-model openai/gpt-oss-20b \
    --reranker-base-url http://localhost:44713/v1 --first-stage-k 50 \
    --candidate-max-tokens 512 --prompt-template-path reasonrank_template_low.yaml

python -m pyserini.eval.trec_eval -c -m recall.5,10 -m ndcg_cut.5,10 \
    topics-qrels/qrel_evidence.txt retrieval_output/<run>.trec

End-to-end deep-research with reranking (Tables 2/4, Figures 2/3)

python search_agent/oss_client.py \
    --model openai/gpt-oss-20b --model-url http://localhost:34713/v1 \
    --searcher-type faiss --index-path "indexes/qwen3-embedding-8b/corpus.shard*.pkl" \
    --model-name Qwen/Qwen3-Embedding-8B --normalize --k 5 --snippet-max-tokens 512 \
    --reranker-type batch_listwise_vllm --reranker-model openai/gpt-oss-20b \
    --reranker-base-url http://localhost:44713/v1 \
    --first-stage-k 20 --candidate-max-tokens 512 --reasoning-token-budget 2048 \
    --prompt-template-path reasonrank_template_low.yaml \
    --reasoning-effort high --max-tokens 16384 --num-threads 32 \
    --output-dir runs/qwen3-embedding-8b/gpt-oss-20b/rerank_l_k_20_search_high \
    --invocation-history-dir runs/qwen3-embedding-8b/gpt-oss-20b/rerank_l_k_20_search_high/invocation_history

Sweeps used in the paper

• Reranking depth: --first-stage-k ∈ {10, 20, 50}; the no-rerank baseline omits all --reranker-* flags.
• Search reasoning: --reasoning-effort ∈ {low, medium, high} with --max-tokens 2048 / 8192 / 16384.
• Heterogeneous cross-encoder (Table 4 / Fig 3): --reranker-type batch_pointwise_vllm --reranker-model Qwen/Qwen3-Reranker-0.6B (no prompt template / reasoning budget needed).
• Agent size: swap --model / --reranker-model between openai/gpt-oss-20b and openai/gpt-oss-120b.

Output is one JSON per query under --output-dir plus reranker call logs under --invocation-history-dir (needed for token accounting). Next: evaluate-and-etc.