| name | ide_jetro |
| description | Platform rules, Taiwan title filter, date extraction, and lookback window for the IDE-JETRO scraper |
| applyTo | scraper/sources/ide_jetro.py |
ide_jetro Scraper — Platform Reference
Platform Profile
| Field | Value |
|---|
| Site URL | https://www.ide.go.jp |
| Listing URL | https://www.ide.go.jp/Japanese/Event/Seminar.html |
| Rendering | Static HTML — uses requests + BeautifulSoup (no Playwright) |
| Auth required | No |
| Rate limit | REQUEST_DELAY = 1.0 s between detail-page fetches |
| Source name | ide_jetro |
| Source ID format | ide-jetro_{YYMMDD} from URL path (e.g. /Japanese/Event/Seminar/250926.html → ide-jetro_250926) |
Field Mappings
| Event Field | Source |
|---|
source_id | ide-jetro_{YYMMDD} from <a href> path |
raw_title | og:title meta tag on detail page, or <h1> |
start_date | First YYYY.MM.DD occurrence in listing <span class="date"> |
end_date | Second date in range notation YYYY.MM.DD〜YYYY.MM.DD; None if single-day |
location_name | IDE venue (usually 幕張メッセ or 研究所内) from body |
raw_description | Detail page body text |
Structured fields required from detail page
For seminar pages, scraper must parse heading-based sections in detail HTML and map them explicitly:
| Detail section | Target field | Rule |
|---|
開催日程 | business_hours | extract time range (HH:MM〜HH:MM or HH時MM分〜HH時MM分) |
会場 | location_name | keep venue text; online webinars should stay オンライン |
主催 | organizer | use exact text from section block |
講師・プログラム table | performers (via annotator) | collect speaker names from speaker column and inject into raw_description |
Do not rely on one generic paragraph-only description. If these sections are not parsed, front-end detail pages lose organizer/speaker/hours.
IDE CMS Block Structure Guard
IDE pages are rendered with nested blocks (pbNested, paragraph, table-basic) after <h4> headings. Traditional sibling-only scraping can return empty strings even when content exists.
Required pattern:
- Locate heading (
開催日程, 会場, 主催, 講師・プログラム).
- Find the first nested content block after that heading (
div with class containing pbNested|paragraph|table-basic).
- Extract plain text / table cells from that block.
When this guard is violated, typical symptoms are:
business_hours = nullorganizer = nullperformer(s) = null
for events that clearly have these fields on the source page.
Raw Description Structure Rule
Always prepend structured lines before free-text summary when values exist:
会場: ...主催: ...時間: ...講師: ...
This ensures annotator can deterministically recover organizer and performer fields.
Taiwan Title Filter
The listing contains hundreds of seminars; only keep items whose title contains 台湾:
if "台湾" not in item_title:
continue
This is the primary relevance gate. Do not widen it — IDE-JETRO events are academic/business and the false-positive rate for non-Taiwan content is high.
Lookback Window
LOOKBACK_DAYS = 180 — only ingest events whose start_date is within the past 180 days or any future date. The listing page accumulates past years; without this cutoff the scraper re-ingests stale records on every run.
Date Format
Listing page dates are YYYY.MM.DD (dotted, ASCII digits):
Single : 2025.09.26 (金曜)
Range : 2025.11.25 (火曜)〜2026.03.13 (金曜)
_parse_date_dotted(s) extracts the first YYYY.MM.DD occurrence.
Source Overlap Note
The /Sympo/ page mirrors the /Seminar/ listing — 100% overlap. Only scrape Seminar.html; do not add the Symposia URL to avoid duplicate ingestion.
Pending Rules
