| name | literature-live-session-pipeline |
| description | Orchestrate lawful literature collection through live Microsoft Edge sessions across ScienceDirect, CNKI, and Web of Science, then create an EndNote X9 library, import RIS metadata, and group records by section_paths from a manifest CSV. Use when browser login or bot verification blocks direct fetching, when the user wants separate browser sessions per source, or when a paper batch must be downloaded into a folder and mirrored into a grouped EndNote library. |
Literature Live Session Pipeline
Use this skill when the browser session is the source of truth and the batch needs both files and reference-management structure.
Quick Start
-
Prepare the manifest CSV.
Read references/input-schema.md.
The core columns are number, title, doi, source_hint, entry_url, candidate_urls, and section_paths.
If the source document is a Word file with EndNote EN.CITE fields, use scripts/extract_manifest_from_endnote_docx.py first.
-
Launch a source-specific Edge session.
Use scripts/launch_edge_live_session.ps1.
Keep separate ports for sciencedirect, cnki, and wos.
-
Let the user finish the manual browser step.
They must sign in lawfully, pass verification pages, open one representative record, and keep the window open.
-
Probe the current page before the full batch.
Use scripts/probe_live_session.py.
If the probe still reports a challenge page, stop and let the user finish the browser step manually.
-
Create the local EndNote library input.
Use scripts/build_ris_from_manifest.py to generate RIS.
Use scripts/create_endnote_library.py if a fresh library is needed.
-
Import the RIS and group records in EndNote.
Use scripts/endnote_pipeline.py.
It can create the library, import RIS, and assign records to flat groups named from section_paths.
-
Fetch the PDFs.
Route CNKI rows to scripts/run_cnki_pdf_live_fetch.ps1. This route opens the exact-title detail page and clicks only the visible PDF下载 control; it never selects CAJ.
Route other publishers to scripts/run_devtools_multi_source_fetch.ps1, which wraps scripts/devtools_multi_source_serial_fetch.py.
Commands
Launch a ScienceDirect browser session:
powershell -ExecutionPolicy Bypass -File "$HOME\.codex\skills\literature-live-session-pipeline\scripts\launch_edge_live_session.ps1" -Source sciencedirect
Launch a CNKI browser session:
powershell -ExecutionPolicy Bypass -File "$HOME\.codex\skills\literature-live-session-pipeline\scripts\launch_edge_live_session.ps1" -Source cnki
Probe a live session:
python "$HOME\.codex\skills\literature-live-session-pipeline\scripts\probe_live_session.py" --debug-port 9222 --url "https://example.com/article"
Build RIS from the manifest:
python "$HOME\.codex\skills\literature-live-session-pipeline\scripts\build_ris_from_manifest.py" `
--input-csv C:\path\to\manifest.csv `
--ris-out C:\path\to\batch\manifest.ris `
--json-out C:\path\to\batch\manifest.records.json
Extract a manifest from an EndNote-embedded DOCX:
python "$HOME\.codex\skills\literature-live-session-pipeline\scripts\extract_manifest_from_endnote_docx.py" `
--docx C:\path\to\manuscript.docx `
--manifest-out C:\path\to\batch\manifest.csv `
--report-json C:\path\to\batch\manifest.extract.json
Create or update the EndNote library:
python "$HOME\.codex\skills\literature-live-session-pipeline\scripts\endnote_pipeline.py" `
--manifest-csv C:\path\to\manifest.csv `
--ris-path C:\path\to\batch\manifest.ris `
--library-name "UHS_THMD_batch" `
--library-root C:\path\to\batch `
--report-json C:\path\to\batch\endnote_report.json
Run the download batch:
powershell -ExecutionPolicy Bypass -File "$HOME\.codex\skills\literature-live-session-pipeline\scripts\run_devtools_multi_source_fetch.ps1" `
-InputCsv C:\path\to\manifest.csv `
-OutDir C:\path\to\batch\downloads `
-DebugPort 9222 `
-InterItemSleepSeconds 6
Run a CNKI PDF-only batch after the user has completed login or any visible verification:
powershell -ExecutionPolicy Bypass -File "$HOME\.codex\skills\literature-live-session-pipeline\scripts\run_cnki_pdf_live_fetch.ps1" `
-InputCsv C:\path\to\cnki-manifest.csv `
-OutDir C:\path\to\batch\cnki-downloads `
-DebugPort 9223 `
-ReuseExistingPage `
-InterItemSleepSeconds 7
Operational Notes
- Treat
Web of Science as a record router, not a guaranteed PDF host. Expect it to resolve DOI or publisher links first.
- For CNKI, use the dedicated PDF-only route: search result -> exact-title detail page -> exact visible text
PDF下载. Do not use element IDs because CNKI can assign the same id="cajDown" to both CAJ and PDF anchors.
- CNKI keeps dormant CAPTCHA markup on normal pages. A prompt found only in raw HTML or body text is not proof of a real slider; require a verification URL/title or a rendered prompt that intersects the viewport.
- CNKI may write
_blank downloads to the Windows Downloads folder even after a batch download directory is configured. The CNKI route watches both locations and copies, rather than removes, a valid system-download source.
- Treat a PDF permissions-encryption flag as metadata, not automatic failure. Accept it when the file opens without a password, has at least two readable pages, and matches the expected title.
- Keep separate Edge user-data directories by source. This is the default behavior of the launch script and is the recommended browser split.
- Use
section_paths to name EndNote groups. Separate multiple groups with |.
- Prefer retrying only failed rows by trimming the manifest rather than re-running the entire batch immediately.
Guardrails
- Stay inside the user's authorized browser session. Do not try to bypass paywalls, verification pages, or access controls.
- Stop if the page remains on a challenge, CAPTCHA, or institutional login screen.
- Do not execute or simulate a slider solution. The workflow only distinguishes a genuinely visible challenge from dormant hidden markup, then pauses for the user.
- Accept CNKI output only when its first bytes are
%PDF-; reject CAJ and other payloads even if the filename ends in .pdf.
- Expect some
WOS and CNKI rows to require manual correction of entry_url or candidate_urls.
- EndNote automation assumes
EndNote X9 on Windows and a visible desktop session.
References
