researcher

name	researcher
description	Source evaluation rules and discovery criteria for the Researcher agent
applyTo	.github/agents/researcher.agent.md

Researcher Skills

Read this at the start of every session before evaluating any source.

⚠️ CRITICAL: Canonical File Path

NEVER write to .github/skills/researcher/ — that path has been deleted. The canonical location is .github/skills/agents/researcher/. All agent skills live under skills/agents/<agent-name>/. See engineer SKILL.md for the full table.

⚠️ CRITICAL: Geographic Scope

Scope is ALL OF JAPAN（全日本） — Tokyo, Osaka, Kyoto, Fukuoka, Nagoya, Sapporo, and all other prefectures are in scope.

FORBIDDEN: Do NOT use "開催地が東京ではない" or "東京スコープ外" as a reason to mark a source not-viable. This is an incorrect justification.

• A source covering only one region (e.g. 福岡のみ) is still researched/viable if it reliably surfaces Taiwan-related events there.
• When profiling, note which region(s) events are held in (region: 全国 | 東京 | 大阪 | 福岡 | ...).
• The question is: does the source have Taiwan events? Not: is the venue in Tokyo?

ENFORCE: card_selector_hint required when feasibility=easy

Before update_source.py --status researched --feasibility easy, self-verify:

Check	Required
Fetched listing page HTML (fetch_webpage / curl)?	✅
Identified the repeating card element from DOM?	✅
Selector is verbatim from HTML (no guess)?	✅
Selector matches ≥1 element in sample HTML?	✅

Failure modes:

• No stable repeating selector → downgrade to --feasibility medium + record reason in --notes. Never pass a fabricated selector.
• Listing requires JS rendering and only static HTML available → downgrade to medium.

Data point (2026-05-02 batch e2e): Phase 2 success rate is 17% (1/6) without a verbatim selector hint — GPT-4o fabricates plausible-looking CSS classes that don't exist in the real DOM.

Source Evaluation

• Check robots.txt and ToS before profiling a scraping target.
• Verify whether the site uses client-side rendering — determines Playwright vs. simple HTTP fetch. Check by viewing page source vs. rendered output.
• Test rate limits: attempt 3 rapid requests and observe response headers (Retry-After, X-RateLimit-*, Retry-After).
• Confirm Taiwan-related events actually appear (not just Japanese-domestic events) before profiling.

Output

• Save source profiles to .copilot-tracking/research/sources/{source_name}.md.
• Always include a scraping feasibility verdict: Easy / Medium / Hard / Blocked.
• Include at least 2 example event URLs so the Scraper Expert can verify selectors.

Research Source Status Values

When writing a source record to the research_sources table, use these status values:

Status	Meaning	When to use
not-viable	Evaluated; will NOT be scraped	Technically unscrapable (login/robots/terminated); or confirmed zero Taiwan events after thorough history check
researched	Deep research complete; ready for scraper issue	Taiwan events confirmed; selectors verified; profile written

pending and viable are NOT valid statuses — update_source.py only accepts not-viable and researched.

Mandatory fields for every insert:

{
    'name': '<source display name>',
    'url': '<canonical URL>',
    'category': '<government|ngo|community|commercial|...>',
    'status': '<not-viable|researched>',
    'reason': '<one sentence why this status>',  # required for not-viable
    'url_verified': True,
    'first_seen_at': now_iso,
    'last_seen_at': now_iso,
}

not-viable reason examples (valid):

• robots.txt がクローラーを明示的に禁止
• ログイン必須でスクレイピング不可
• サービス終了済み（例：PassMarket）
• 過去のアーカイブを徹底調査した結果、台湾関連活動がゼロ件
• ストリーミング/VODサービス（物理的イベントが存在しない）

NOT valid not-viable reasons:

• 「東京以外で開催」 ← スコープは全日本。絶対にこれを理由にしない。
• 「台湾イベントが年1-2件以下で少ない」 ← 低頻率は理由にならない（下記 Low-Signal Policy 参照）
• 「現在は台湾コンテンツなし」 ← 過去実績があれば viable（将来も出る可能性）

Low-Signal Source Policy（2026-04-27）

The scraper runs once per day. Adding an extra source costs near-zero CPU. Optimize for coverage over precision — missing a signal is worse than scanning 0 events.

New acceptance threshold: A source is viable if:

1. Taiwan-related events have ever occurred in its history, AND
2. The source is technically scrapable (Playwright OK; no login required; not terminated)

"Too infrequent" is no longer a valid rejection reason.

Low-Signal Policy — assessed sources (score 0.30–0.69)

auto_research marks these assessed and leaves them for human review. Human reviewers MAY manually set status=researched when:

• Daily scraping cost is near-zero (Playwright static HTML, < 5 pages)
• Missing a rare Taiwan event is costlier than scanning 0-result pages
• The site hosts annual events / film festivals whose schedule is not predictable in advance (e.g. JPIFF, ACROS Fukuoka, Internet Museum)
• LLM found indirect or low-signal evidence (Taiwan artists "included", Taiwan film "sometimes shown")

NOT a valid reason to keep assessed:

• 「スコアが低い」 ← Score < 0.70 は「台湾専用ではない」を示すだけ
• 「現時点でコンテンツなし」 ← 将来の掲載可能性があれば viable

Scale for decision:

score	taiwan_evidence	Action
≥ 0.70	Direct	auto-promoted → researched
0.30–0.69	Indirect / low	Human review → manually set researched if scrapable
< 0.30	None	Keep not-viable

Use LOOKBACK_DAYS to match the source's natural cadence:

Source cadence	Recommended LOOKBACK_DAYS
Weekly / daily	30
Monthly	60
Quarterly	90
Annual	365
Biennial	730

Re-evaluation candidates (previously marked not-viable for frequency/Tokyo-scope reasons):

• [27] 京都大学人文科学研究所 — scope exclusion lifted
• [81] 福岡アジア美術館 — scope exclusion lifted
• [41] シネスイッチ銀座 — occasional Taiwan films viable with daily scan
• [38] Uplink Shibuya — proven Taiwan film history
• [35] Human Trust Cinema — re-verify URL
• [1] 東大先端研 (RCAST) — Taiwan × economic security events
• [2] Asia University Asian Studies — Asia Watcher series
• [78] note.com — curated creator list approach

researcher.py — Schedule Management (SLOT_SCHEDULE)

researcher.py uses a 4-slot daily schedule to cover all 9 research categories every day.

Slot	JST	Categories
0	06:00	university, fukuoka
1	12:00	media, government
2	18:00	thinktank, hokkaido
3	00:00	social, performing_arts_search, senses_research

Key implementation rules:

• RESEARCH_SLOT env var controls which slot runs. Set by GitHub Actions step before calling python researcher.py.
• _resolve_slot(): reads RESEARCH_SLOT env var first; falls back to deriving from JST hour.
• _resolve_category_id() returns list[str] — run_research() loops over each category in the slot.
• GitHub Actions: 4 cron triggers (0 21/3/9/15 * * * UTC). A Determine slot step maps UTC hour → slot and writes to $GITHUB_ENV.
• github.event.schedule returns the exact cron string (e.g. "0 21 * * *") for the triggering schedule. Use shell case to compare it and set RESEARCH_SLOT.
• workflow_dispatch sets github.event.schedule to empty string — use inputs.slot as fallback.

Duplicate suppression in LINE reports:

• known_urls is fetched before agents run (pre-run snapshot). After _upsert_sources() completes, filter report["top_sources"] to remove any URL already in known_urls:

  report["top_sources"] = [s for s in report["top_sources"] if s.get("url") not in known_urls]
  

• If no new verified sources remain after filtering, skip LINE notification entirely (log only). This prevents notification fatigue on days when GPT returns only already-known sources.
• scraper_runs cost logging and research_reports DB save are unaffected by this filter.

When modifying the slot-to-category mapping, update both SLOT_SCHEDULE in researcher.py and the corresponding cron comments in researcher.yml.

Domain-level skip_hint (Fix A — commit ab46560):

• CategoryAgent.run() の skip_hint は exact URL ではなく 正規化ドメインのリスト で渡す
• 実装パターン:

  skip_domains = {urlparse(u).netloc.lstrip("www.") for u in known_urls}
  skip_hint = "SKIP these already-covered domains: " + ", ".join(sorted(skip_domains))
  

• 理由：exact URL では同一ドメインの別パスが毎日新候補として生成され not-viable レコードが無限増加する

Candidate domain dedup (Fix B — commit ab46560):

• _upsert_sources() のドメイン dedup は candidate ステータスを含む 全ステータス でブロックする
• www.asahi.com と asahi.com を別候補として挿入させない
• ブロック判定: normalize_domain(existing_url) == normalize_domain(new_url) で全 DB 行を確認

discovery_accounts.py — Layer 2 (Weekly note.com Creator Discovery)

scraper/discovery_accounts.py discovers new note.com creators who post Taiwan-related content.

Workflow (runs every Sunday 10:00 JST):

1. 3 GPT gpt-4o-search-preview tasks (community events / culture & arts / food & lifestyle)
2. _extract_creator_id() normalises GPT output → bare slug; rejects article URLs (/n/) and template patterns
3. _verify_note_creator() confirms existence via https://note.com/{creator_id}/rss (HTTP GET only — no Playwright needed)
4. Loads existing known IDs from research_sources to skip re-insertion
5. Upserts verified creators with status='candidate', source_profile={platform: 'note.com', creator_id, categories: ['taiwan_japan']}
6. Admin reviews at /admin/sources and sets status='implemented' to activate Layer 3 scraping

Key rules:

• note.com RSS: https://note.com/{creator_id}/rss is publicly accessible without auth.
• GPT output often includes article URLs (/n/ path) and template strings — always validate with _extract_creator_id().
• Use --dry-run for testing: runs GPT + verification but skips DB writes and LINE push.
• The daily scraper (note_creators.py or similar Layer 3 script) polls only status='implemented' creators from research_sources.
• 年份は _THIS_YEAR = datetime.now(JST).year を使う。 search query 文字列に年数を hardcode してはならない（毎年手動更新が必要になり、古い検索結果しか返らなくなる）。

WordPress Mixed-Content Sites（日本／台湾イベント混在サイト）

For WordPress sites that publish both Japan-hosted and Taiwan-hosted events:

1. Listing-page date pre-filter (90-day window) Before fetching any article, parse <time datetime="..."> on the listing page to get the post publish date. Skip articles older than 90 days and stop paginating when an entire page is older than 90 days. This can reduce HTTP requests by 30–40× (e.g. 220 → 6 fetches).

2. Three-pass Japan-event filter (_is_japan_event()) Apply in this exact order — Stage 2 must come before Stage 3:

• Stage 1 (title): TAIWAN_ONLY_PATTERNS — exclude events clearly held in Taiwan (e.g. 台湾ランタン, 澎湖花火)
• Stage 2 (body): TAIWAN_VENUE_KW — exclude if venue is explicitly in Taiwan (（台湾・, （台湾）, etc.)
• Stage 3 (body): JAPAN_LOCATION_KW — include only if a Japan city/region name is present

Reversing Stage 2 and Stage 3 causes false positives: a Taiwan-held event can mention Japanese travel companies (e.g. 近畿日本ツーリスト), which triggers Stage 3 before Stage 2 can reject it.

3. Date extraction priority ladder for Japanese WordPress Never rely on the first date in the body — it is almost always the post publish date. Use this priority:

1. 日時： labeled date range
2. Weekday-annotated range (e.g. YYYY年M月D日（土）〜D日（日）)
3. Labeled single date
4. Weekday-annotated single date
5. Plain date range
6. Last resort: first plain date in body (high risk of picking up the publish date)

4. WordPress REST API check first Always try /wp-json/wp/v2/posts first. If 401, fall back to HTML scraping.

Manual Source DB Insert Workflow

update_source.py --create-issue performs an UPDATE only — it requires the row to already exist in research_sources. When a source is discovered manually (not via researcher.py), insert the row first:

sb.table("research_sources").insert({
    "name": "<display name>",
    "url": "<canonical URL>",
    "category": "<government|ngo|community|commercial|...>",
    "status": "candidate",
    "reason": "<one sentence>",
    "url_verified": True,
    "first_seen_at": now_iso,
    "last_seen_at": now_iso,
}).execute()

Note: the column is reason, not notes (that column does not exist). After INSERT, run: python scraper/update_source.py --url <url> --status researched --create-issue

Events テーブル — URL フィールド分離規則（2026-05-07）

3 つの URL フィールドは役割が異なる。混在させない。

フィールド	役割	上書き可否
source_url	爬蟲が取得した原始 URL（戲院ページ/ニュース記事）	絶対不可（NOT NULL 制約、null 不可）
official_url	配給会社/公式サイト（全国上映情報など）	新規追加・更新はここ
secondary_source_urls	合併元の追加 URL（配列）	source_url と重複させない

操作手順（公式サイト URL を追加する場合）:

sb.table("events").update({"official_url": "https://example.com/"}).eq("id", eid).execute()
# NG: source_url は触らない

secondary_source_urls の重複排除:

# source_url と同じ URL を secondary から除外してから保存
secondary = [u for u in secondary_urls if u != event["source_url"]]

Works テーブル — 検索・新規作成規則（2026-05-07）

検索時は全タイトル列 + director を OR 検索すること:

SELECT * FROM works
WHERE title_ja ILIKE '%keyword%'
   OR title_zh ILIKE '%keyword%'
   OR original_title ILIKE '%keyword%'
   OR title_en ILIKE '%keyword%'
   OR director ILIKE '%keyword%';

events 検索も raw_title と name_ja の両方を確認する（annotation 後に title が変わる）:

SELECT * FROM events
WHERE raw_title ILIKE '%keyword%'
   OR name_ja ILIKE '%keyword%';

新規 work 作成時の必須フィールド:

{
    "title_ja": "<日本語タイトル>",
    "title_zh": "<繁体字タイトル>",
    "title_en": "<英語タイトル（公式 og:title など）>",
    "director": "<中文漢字（読み）/ 中文漢字（読み）>",  # 共同監督はスラッシュ区切り
    "external_links": {"official": "<公式URL>"},
}

監督名フォーマット例: 黄明川（ホアン・ミンツァン）/ 連楨惠（リェン・チェンフイ）

After a Source Evaluation Error

1. Append an entry to .github/skills/agents/researcher/history.md (newest at top).
2. If the lesson generalizes, add a rule to this file.

--create-issue Permission Consistency

When documenting or running source status updates with python scraper/update_source.py --create-issue, use one canonical permission wording:

• Fine-grained PAT: Issues: write + Metadata: read
• Classic token: repo scope

Do not use non-standard expressions (such as &-separated permissions) in profiles, summaries, or agent guidance.

Token Checklist Source of Truth

For token sync and rotation references, always point to:

• docs/GITHUB_TOKEN_SYNC_CHECKLIST.md (canonical checklist)

/.github/TOKEN_SYNC_CHECKLIST.md is redirect-only and should not be used as an editable checklist body.