القائمة
Harvest metadata from open repositories using OAI-PMH protocol
يُحدَّد Prompt الذي يراجع المصدر أولًا بشكل افتراضي. يمكنك التبديل إلى أمر مباشر أو تنزيل نسخة محلية.
اقرأ SKILL.md وأي ملفات مرافقة يعرضها SkillsMP قبل أن تقرر التثبيت.
التثبيت باستخدام Codex أو Claude انسخ هذا Prompt والصقه في Codex أو Claude أو مساعد آخر ليراجع صفحة Skill ويثبّتها لك.
يتجاوز الأمر المباشر Prompt المخصّص للمراجعة. افحص المصدر قبل تشغيله.
npx skills add https://github.com/brycewang-stanford/Auto-Empirical-Research-Skills --skill repository-harvesting-guideيبقى الأمر في سطر واحد. مرّر أفقيًا لمراجعته كاملًا قبل النسخ.
تفضّل نسخة محلية؟ نزّل الملفات المتاحة حاليًا لدى SkillsMP.
Route empirical-research requests through the Auto-Empirical Research Skills catalog when this whole repository is installed as one skill in Codex, CodeBuddy, Claude Code, or another IDE. Use to choose and load the right vendored AERS skill for causal inference, econometrics, replication, data acquisition, manuscript writing, peer review and referee responses, citation checking, de-AIGC editing, or full empirical-paper workflows without reading the entire repository at once.
中英双语学术降 AIGC / bilingual academic de-AIGC skill. Removes AI-generated writing signatures from empirical papers in economics, management, and the social sciences — in both English and Chinese. Covers Turnitin AI, GPTZero, Originality.ai on the English side and 知网 AMLC, 万方, 维普 on the Chinese side. Uses a six-step loop (intake → audit → claim-evidence check → differentiated rewrite → five-dimension self-score → cold-reader recheck) with two pattern libraries (22 English + 17 Chinese patterns), section-by-section strategies for empirical papers, and hard protections that keep every number, coefficient, and citation intact.
Use when a research task needs reproducible Kaggle discovery, metadata inspection, bounded public-data downloads, competition or kernel discovery, model discovery, or an explicitly approved Kaggle write/delete operation through the official CLI.
استنادا إلى تصنيف SOC المهني
عرض SKILL.md
| name | repository-harvesting-guide |
| description | Harvest metadata from open repositories using OAI-PMH protocol |
| metadata | {"openclaw":{"emoji":"🚜","category":"tools","subcategory":"scraping","keywords":["OAI-PMH","metadata harvesting","open repositories","Dublin Core","institutional repositories","data providers"],"source":"wentor-research-plugins"}} |
A skill for harvesting metadata from open access repositories using the OAI-PMH (Open Archives Initiative Protocol for Metadata Harvesting) protocol. Covers protocol fundamentals, building harvesters in Python, handling resumption tokens for large collections, metadata format parsing (Dublin Core, MARC, METS), selective harvesting by date and set, and integrating harvested data into research workflows.
OAI-PMH is a standardized protocol that allows metadata to be harvested from repository systems. It is the backbone of library interoperability and is supported by virtually every institutional repository, preprint server, and digital library worldwide.
OAI-PMH Architecture:
Data Providers (repositories):
- Expose metadata through a standardized HTTP interface
- Must support Dublin Core as minimum metadata format
- May support additional formats (MARC, MODS, DataCite, etc.)
- Examples: arXiv, PubMed Central, DSpace repositories,
EPrints, institutional repositories
Service Providers (harvesters):
- Send HTTP requests to data providers
- Collect, aggregate, and index metadata
- Build search services, union catalogs, analytics
- Examples: BASE (Bielefeld), CORE, OpenDOAR
Protocol Version: 2.0 (current, since 2002)
Transport: HTTP GET or POST
Response format: XML
Base URL example: https://arxiv.org/oai2
OAI-PMH defines exactly six request types (verbs):
1. Identify
Purpose: Describe the repository
URL: baseURL?verb=Identify
Returns: repository name, admin email, earliest datestamp,
granularity, compression support
2. ListMetadataFormats
Purpose: List available metadata formats
URL: baseURL?verb=ListMetadataFormats
Returns: format prefixes (oai_dc, marc21, datacite, etc.)
Optional: identifier parameter to check formats for one record
3. ListSets
Purpose: List available sets (collections/categories)
URL: baseURL?verb=ListSets
Returns: set names and specs for selective harvesting
Example sets: physics:hep-th, cs:AI, math:AG
4. ListIdentifiers
Purpose: List record identifiers (headers only, no metadata)
URL: baseURL?verb=ListIdentifiers&metadataPrefix=oai_dc
Optional: from, until, set parameters
Returns: identifiers, datestamps, set memberships
5. ListRecords
Purpose: Harvest full metadata records
URL: baseURL?verb=ListRecords&metadataPrefix=oai_dc
Optional: from, until, set parameters
Returns: complete metadata records in requested format
6. GetRecord
Purpose: Retrieve a single record by identifier
URL: baseURL?verb=GetRecord&identifier=oai:arxiv:2301.00001
&metadataPrefix=oai_dc
Returns: one complete metadata record
import requests
import xml.etree.ElementTree as ET
import time
OAI_NS = "http://www.openarchives.org/OAI/2.0/"
DC_NS = "http://purl.org/dc/elements/1.1/"
def harvest_records(base_url, metadata_prefix="oai_dc",
from_date=None, until_date=None,
set_spec=None):
"""
Harvest all records from an OAI-PMH endpoint.
Handles resumption tokens for paginated results.
Args:
base_url: OAI-PMH base URL
metadata_prefix: metadata format (default: oai_dc)
from_date: selective harvest start (YYYY-MM-DD)
until_date: selective harvest end (YYYY-MM-DD)
set_spec: restrict to a specific set
"""
params = {
"verb": "ListRecords",
"metadataPrefix": metadata_prefix,
}
if from_date:
params["from"] = from_date
if until_date:
params["until"] = until_date
if set_spec:
params["set"] = set_spec
all_records = []
request_count = 0
while True:
response = requests.get(base_url, params=params, timeout=30)
response.raise_for_status()
request_count += 1
root = ET.fromstring(response.content)
# Parse records from this page
records = root.findall(
f".//{{{OAI_NS}}}record"
)
for record in records:
parsed = parse_dublin_core(record)
if parsed:
all_records.append(parsed)
# Check for resumption token
token_elem = root.find(
)
token_elem token_elem.text:
params = {
: ,
: token_elem.text,
}
time.sleep()
:
(
)
all_records
():
header = record_element.find()
metadata = record_element.find()
header metadata :
status = header.get(, )
status == :
identifier = header.findtext(, )
datestamp = header.findtext(, )
dc = metadata.find()
result = {
: identifier,
: datestamp,
: find_dc_text(metadata, ),
: find_dc_all(metadata, ),
: find_dc_all(metadata, ),
: find_dc_text(metadata, ),
: find_dc_text(metadata, ),
: find_dc_text(metadata, ),
: find_dc_all(metadata, ),
: find_dc_text(metadata, ),
: find_dc_text(metadata, ),
}
result
():
elem = metadata.find()
elem.text elem
():
elems = metadata.findall()
[e.text e elems e.text]
Incremental harvesting strategy:
First harvest: Get everything
from_date = None (or repository's earliestDatestamp)
until_date = today
Subsequent harvests: Get only new/modified records
from_date = last_harvest_date
until_date = today
Date granularity:
- Day-level: YYYY-MM-DD (most common)
- Second-level: YYYY-MM-DDThh:mm:ssZ (some repositories)
- Check the Identify response for supported granularity
Important: OAI-PMH datestamps reflect the date the METADATA
was last modified, not the publication date. A record edited
yesterday to fix a typo will appear in a harvest with
from=yesterday, even if the paper was published in 2015.
Common set structures by repository type:
arXiv:
physics, physics:hep-th, cs, cs:AI, math, math:AG, etc.
DSpace repositories:
com_12345_1 (community), col_12345_2 (collection)
Hierarchical: department -> collection
PubMed Central:
By journal: pmc-journal-name
By funder: pmc-funder-name
Strategy:
1. Call ListSets to see available sets
2. Identify sets relevant to your research topic
3. Harvest only those sets to reduce data volume
4. Store the set membership for each record
Quality problems in harvested metadata:
1. Duplicate records:
- Same paper in multiple repositories
- Same paper in multiple sets within one repository
- Solution: Deduplicate by DOI, then by title similarity
2. Incomplete metadata:
- Missing abstracts (very common)
- Missing author identifiers
- Missing dates or using inconsistent date formats
- Solution: Enrich with Crossref or OpenAlex lookups
3. Encoding issues:
- Non-UTF-8 characters in older repositories
- HTML entities in text fields
- Solution: Normalize encoding, strip HTML tags
4. Inconsistent formats:
- Dates as "2023", "2023-01", "2023-01-15", "January 2023"
- Author names as "Smith, John" vs "John Smith" vs "J. Smith"
- Solution: Parse and normalize to canonical formats
Major repositories with OAI-PMH support:
arXiv: https://export.arxiv.org/oai2
PubMed Central: https://www.ncbi.nlm.nih.gov/pmc/oai/oai.cgi
Europeana: https://oai.europeana.eu/oai
HAL (France): https://api.archives-ouvertes.fr/oai/hal
DBLP: https://dblp.org/oai
CiteSeerX: https://citeseerx.ist.psu.edu/oai2
To find more endpoints:
- OpenDOAR directory: https://v2.sherpa.ac.uk/opendoar/
- ROAR (Registry of Open Access Repositories)
- BASE (Bielefeld Academic Search Engine) source list
OAI-PMH harvesting remains the most reliable method for building comprehensive metadata collections from open repositories. While newer APIs like ResourceSync and Signposting offer richer functionality, OAI-PMH's universal adoption and simplicity make it the practical choice for most academic metadata collection tasks.