omnidocbench-eval-helper

# OmniDocBench evaluation helper Help community users run OmniDocBench reproducibly. Prefer a practical, path-specific workflow over generic setup advice: 1. Validate GT/prediction paths and runtime access. 2. Generate an isolated config using stable container paths. 3. Run `python pdf_validation.py --config ...` in Docker when possible. 4. Parse `*_metric_result.json`, `*_run_summary.json`, `*_stage_execution.json`, and `*_runtime_environment.json`. 5. Report scores, output paths, and any warnings that affect score credibility. Be concise for simple questions, but include copy-paste commands when the user gives paths. ## What to ask for Ask only for missing details that affect commands or config: - Deployment mode: Docker, conda/source, or remote SSH. Recommend Docker when CDM is needed. - Evaluation type: default to `end2end` for OmniDocBench JSON + page markdown predictions. - Paths: - ground-truth JSON, commonly `OmniDocBench.json` with exact capitalization on Linux - prediction markdown directory, often a nested `markdown/` folder such as `.../mineru/markdown` - output directory, or permission to create a timestamped output directory - Whether CDM is required. - CPU/RAM or node size if the user mentions slowness, OOM, or cluster execution. If the user explicitly asks you to run a job on a remote host and provides an SSH command, treat that as authorization for that host/path scope. Avoid destructive operations: do not delete, overwrite, or chmod shared data/results. ## Key facts - Entrypoint: `python pdf_validation.py --config <config_path>`. - Default config: `configs/end2end.yaml`. - Package: `omnidocbench-eval`, Python `>=3.10,<3.12`. - Recommended Docker image: `ghcr.io/zeng-weijun/omnidocbench-eval:repro-ubuntu2204`. - CDM depends on TeX Live/CJK, ImageMagick PDF read/write, and Ghostscript. Docker avoids most CDM dependency failures. - Worker keys: - `dataset.match_workers`: page matching - `metrics.display_formula.cdm_workers`: CDM rendering/comparison, memory-heavy - `metrics.table.teds_workers`: table TEDS - Worker rule: on 4 CPU / 8 GB, use `2` for match/CDM/TEDS; if unstable, use `1`. Do not keep README default `13` on small nodes. ## Input validation before running Always validate before launching a long Docker evaluation. For remote runs, execute this on the remote host via SSH. ```bash GT=/path/to/OmniDocBench.json PRED=/path/to/prediction/markdown test -f "$GT" && echo "GT_OK=$GT" || echo "GT_MISSING=$GT" test -d "$PRED" && echo "PRED_OK=$PRED" || echo "PRED_MISSING=$PRED" [ -f "$GT" ] && wc -c "$GT" [ -d "$PRED" ] && echo "md_count=$(find "$PRED" -maxdepth 1 -name '*.md' | wc -l)" [ -d "$PRED" ] && echo "empty_md=$(find "$PRED" -maxdepth 1 -name '*.md' -empty | wc -l)" [ -d "$PRED" ] && find "$PRED" -maxdepth 1 -name '*.md' | sort | head -5 nproc free -h || true ``` Interpretation: - Full OmniDocBench v1.6 has 1651 pages. `md_count` should usually be close to 1651 for a full run. - `md_count=0` usually means the user passed a parent directory instead of the actual `markdown/` directory. - `empty_md > 0` is not fatal, but report it because empty predictions lower scores. - Linux paths are case-sensitive: `OmniDocBench.json` and `omnidocbench.json` differ. If GT is missing, check nearby JSON filenames before concluding the data is absent. ## Docker workflow Use Docker when possible, especially for CDM. ### Check Docker access ```bash DOCKER=docker if ! docker ps >/dev/null 2>&1; then if sudo -n docker ps >/dev/null 2>&1; then DOCKER="sudo -n docker" else echo "Docker daemon is not accessible by docker or sudo -n docker" exit 4 fi fi $DOCKER --version ``` Use `sudo -n docker` only when the user is authorized on that host. Do not try interactive sudo in an automated workflow. ### Local end2end command Use stable container paths and write a custom config inside the container. This avoids editing the repository and keeps host paths out of YAML. ```bash GT=/abs/path/to/OmniDocBench.json PRED=/abs/path/to/prediction/markdown OUT=/abs/path/to/output_dir WORKERS=4 # use 2 on 4CPU/8G; use 1 if CDM OOMs IMAGE=ghcr.io/zeng-weijun/omnidocbench-eval:repro-ubuntu2204 mkdir -p "$OUT" DOCKER=docker if ! docker ps >/dev/null 2>&1; then if sudo -n docker ps >/dev/null 2>&1; then DOCKER="sudo -n docker"; else exit 4; fi fi $DOCKER pull "$IMAGE" $DOCKER run --rm --entrypoint bash \ -v "$GT":/workspace/gt/OmniDocBench.json:ro \ -v "$PRED":/workspace/data_md/predictions:ro \ -v "$OUT":/workspace/result \ "$IMAGE" \ -lc "cat > configs/custom_end2end.yaml <<EOF end2end_eval: metrics: text_block: metric: [Edit_dist] display_formula: metric: [Edit_dist, CDM] cdm_workers: ${WORKERS} table: metric: [TEDS, Edit_dist] teds_workers: ${WORKERS} reading_order: metric: [Edit_dist] dataset: dataset_name: end2end_dataset ground_truth: data_path: ./gt/OmniDocBench.json prediction: data_path: ./data_md/predictions match_method: quick_match match_workers: ${WORKERS} quick_match_truncated_timeout_sec: 300 match_timeout_sec: 420 timeout_fallback_max_chunk_span: 10 timeout_fallback_order_penalty: 0.10 EOF python pdf_validation.py --config configs/custom_end2end.yaml 2>&1 | tee /workspace/result/eval.log" ``` If CDM is not needed, remove `CDM` from `display_formula.metric` and remove `cdm_workers`. This avoids TeX/ImageMagick/Ghostscript dependency failures, but the final report will not include Formula CDM. ## Remote SSH / H-cluster workflow When the user provides an SSH command, run the same validation and Docker workflow on the remote host. Use a heredoc remote script to avoid nested quoting bugs. ```bash ssh -CAXY user@host 'bash -s' <<'REMOTE' set -euo pipefail GT=/mnt/shared-storage-user/.../1.6/OmniDocBench.json PRED=/mnt/shared-storage-user/.../mineru/markdown OUT_ROOT=/mnt/shared-storage-user/.../omnidocbench_eval OUT="$OUT_ROOT/mineru_$(date +%Y%m%d_%H%M%S)" LATEST=/tmp/omnidocbench_latest_out.txt IMAGE=ghcr.io/zeng-weijun/omnidocbench-eval:repro-ubuntu2204 mkdir -p "$OUT" echo "$OUT" > "$LATEST" test -f "$GT" || { echo "GT_MISSING=$GT"; exit 2; } test -d "$PRED" || { echo "PRED_MISSING=$PRED"; exit 2; } MD_COUNT=$(find "$PRED" -maxdepth 1 -name '*.md' | wc -l) EMPTY_MD=$(find "$PRED" -maxdepth 1 -name '*.md' -empty | wc -l) echo "GT_OK=$GT" echo "PRED_OK=$PRED" echo "md_count=$MD_COUNT" echo "empty_md=$EMPTY_MD" [ "$MD_COUNT" -gt 0 ] || { echo "No markdown files found; check nested markdown directory"; exit 3; } DOCKER=docker if ! docker ps >/dev/null 2>&1; then if sudo -n docker ps >/dev/null 2>&1; then DOCKER="sudo -n docker" else echo "Docker daemon is not accessible by docker or sudo -n docker" exit 4 fi fi CPU=$(nproc) MEM_KB=$(awk '/MemTotal/ {print $2}' /proc/meminfo 2>/dev/null || echo 0) MEM_GB=$((MEM_KB / 1024 / 1024)) if [ -n "${FORCE_WORKERS:-}" ]; then WORKERS="$FORCE_WORKERS" elif [ "$CPU" -le 4 ] || [ "$MEM_GB" -lt 10 ]; then WORKERS=2 else WORKERS=4 fi echo "output_dir=$OUT" echo "latest_pointer=$LATEST" echo "cpu=$CPU mem_gb=$MEM_GB workers=$WORKERS" $DOCKER pull "$IMAGE" $DOCKER run --rm --entrypoint bash \ -v "$GT":/workspace/gt/OmniDocBench.json:ro \ -v "$PRED":/workspace/data_md/predictions:ro \ -v "$OUT":/workspace/result \ "$IMAGE" \ -lc "cat > configs/custom_end2end.yaml <<EOF end2end_eval: metrics: text_block: metric: [Edit_dist] display_formula: metric: [Edit_dist, CDM] cdm_workers: ${WORKERS} table: metric: [TEDS, Edit_dist] teds_workers: ${WORKERS} reading_order: metric: [Edit_dist] dataset: dataset_name: end2end_dataset ground_truth: data_path: ./gt/OmniDocBench.json prediction: data_path: ./data_md/predictions match_method: quick_match match_workers: ${WORKERS} quick_match_truncated_timeout_sec: 300 match_timeout_sec: 420 timeout_fallback_max_chunk_span: 10 timeout_fallback_order_penalty: 0.10 EOF python pdf_validation.py --config configs/custom_end2end.yaml 2>&1 | tee /workspace/result/eval.log" echo "DONE_OUT=$OUT" ls -lh "$OUT" REMOTE ``` Practical H-cluster lessons from a real MinerU v1.6 run: - `ssh -CAXY` may print X11 warnings such as `No xauth data`; they are harmless for CLI evaluation. - Docker may be installed but inaccessible through the daemon socket. Check `docker ps`; if it fails and `sudo -n docker ps` works, use `sudo -n docker` for the run. - Record the output directory in `/tmp/omnidocbench_latest_out.txt` or a user-specific variant so it can be recovered after disconnection. - GT capitalization matters: `/.../OmniDocBench.json` worked where lowercase `omnidocbench.json` did not. - MinerU predictions may live in a nested `markdown/` directory. Passing the parent directory can produce zero or inflated markdown counts. - For 4CPU/8G with CDM, start with workers `2`; use `1` if unstable. ## Bundled helper scripts This skill includes scripts that can be copied into the repo or run from the skill directory: - `scripts/generate_end2end_config.py`: generate an end2end YAML. - `scripts/parse_results.py`: parse a result directory and print a compact score/validation report. Example: ```bash python scripts/generate_end2end_config.py \ --gt ./gt/OmniDocBench.json \ --pred ./data_md/predictions \ --out configs/custom_end2end.yaml \ --workers 2 python scripts/parse_results.py /path/to/result_dir ``` Use `--no-cdm` with `generate_end2end_config.py` to omit CDM and `cdm_workers`. ## Conda/source workflow Use only when Docker is unavailable or the user needs to modify OmniDocBench code. ```bash conda create -n omnidocbench python=3.10 -y conda activate omnidocbench pip install -e . python -c "from src.core.pipeline import run_config_file; print('OK')" ``` For CDM in source installs, verify: ```bash pdflatex --version | head -2 kpsewhich CJK.sty && kpsewhich c70gkai.fd gs --version magick --version | head -2 python -m pytest tools/test_environment_and_smoke.py::TestEnvironmentVersions -v -s ``` If ImageMagick blocks PDF read/write, adjust the ImageMagick 7 `policy.xml`. Prefer Docker instead when possible. ## Config guidance ### End2end Use `dataset_name: end2end_dataset`, OmniDocBench JSON as `ground_truth.data_path`, and page-level markdown files as `prediction.data_path`. Use `match_method: quick_match` unless the user has a specific reason not to. Prediction filenames should correspond to page image names with `.jpg`/`.png` replaced by `.md`. Some code paths also accept `.mmd`, but `.md` is the community-standard expectation. ### md2md Use md2md when both ground truth and prediction are markdown directories: ```yaml dataset: dataset_name: md2md_dataset ground_truth: data_path: /path/to/gt_mds page_info: /path/to/OmniDocBench.json prediction: data_path: /path/to/pred_mds match_method: quick_match ``` `page_info` enables page-level attributes and filtering. ### Single-module recognition and detection For formula/text/table recognition, the JSON usually contains both ground-truth and prediction fields; set `prediction.data_key` to the model field such as `pred`. For layout/formula detection, ask for prediction JSON box schema and category names before writing `pred_cat_mapping`. ## Result parsing and validation After a successful run, list result files. Key files usually include: - `*_metric_result.json`: raw metric data and attribute breakdowns - `*_run_summary.json`: notebook-style summary and `overall_notebook` - `*_stage_execution.json`: page count, worker counts, timeout/error/exception counts - `*_runtime_environment.json`: Python/system/TeX/ImageMagick/Ghostscript details - `eval.log`: full console log Extract and report these fields: ```text overall = run_summary.notebook_metric_summary.overall_notebook text_edit = metric_result.text_block.all.Edit_dist.ALL_page_avg formula_cdm = metric_result.display_formula.page.CDM.ALL * 100 formula_edit = metric_result.display_formula.all.Edit_dist.ALL_page_avg table_teds = metric_result.table.page.TEDS.ALL * 100 table_teds_structure = metric_result.table.page.TEDS_structure_only.ALL * 100 table_edit = metric_result.table.all.Edit_dist.ALL_page_avg reading_order_edit = metric_result.reading_order.all.Edit_dist.ALL_page_avg ``` Also report validation counters: - input prediction count and empty markdown count, if checked - `stage_execution.page_match.page_count` - `stage_execution.page_match.fallbacks.quick_match_timeout.count` - `stage_execution.metrics.display_formula.CDM.sample_count`, `timeout_case_count`, `error_case_count`, `exception_case_count`