vendor-scrape

name	vendor-scrape
description	Build-time auto-extraction of LSB-shipped data into compile-time Rust constants. Apply this pattern whenever a Rust crate needs values that LSB defines in a parseable form (headers, SQL, lua tables) — never hand-maintain those values; let cargo regenerate them whenever vendor data updates.
user-invocable	false

vendor-scrape

This codebase already has two precedents for "scrape LSB at build time into Rust statics":

• ffxi-proto/build.rs scrapes vendor/server/src/map/enums/msg_*.h into typed message-ID tables (msg_basic, msg_channel, msg_area, msg_action_modifier, msg_system).
• ffxi-nav/build.rs scrapes vendor/server/sql/zonelines.sql into a static &[ZoneLine] array indexed by from_zone.

Apply this pattern any time you need values from LSB that are defined in parseable form. Hand-maintained mirrors drift silently the moment LSB upstream changes a value — the user has been bitten by this class enough that it's worth always paying the build.rs tax.

Decision: should I scrape, or hand-write?

Scrape when:

• LSB defines the values in a structured source — a C++ header enum, an SQL INSERT INTO, a JSON/YAML, a lua table file
• The values are unlikely to need Rust-side annotation (i.e., we just need the raw value)
• There's a deterministic name mapping from LSB to Rust (the scraper can produce the Rust identifier mechanically)
• The count is more than ~5 entries (under that, the maintenance cost of a scraper outweighs the drift risk)

Hand-write when:

• The values need Rust-side semantics LSB doesn't capture (custom associations, derived traits beyond Copy / PartialEq)
• The LSB source has irregular structure that would make the scraper as long as the table
• There are < ~5 values and they're stable (e.g., a 3-entry protocol-version enum)

Pattern template

A scraper has three pieces:

1. build.rs — emits Rust source to OUT_DIR

// build.rs
use std::{env, fs, path::PathBuf};

const SOURCE: &str = "../vendor/server/src/map/enums/foo.h";

fn main() -> anyhow::Result<()> {
    println!("cargo:rerun-if-changed=build.rs");
    println!("cargo:rerun-if-changed={SOURCE}");

    let src = fs::read_to_string(SOURCE)?;
    let entries = parse_foo_enum(&src)?;

    let mut out = String::from(
        "// AUTO-GENERATED by build.rs from foo.h. Do not edit.\n\n",
    );
    out.push_str("pub const FOO_TABLE: &[(u16, &str)] = &[\n");
    for (id, name) in &entries {
        out.push_str(&format!("    ({id}, {name:?}),\n"));
    }
    out.push_str("];\n");

    let dest = PathBuf::from(env::var("OUT_DIR")?).join("foo_table.rs");
    fs::write(dest, out)?;
    println!("cargo:warning=scraped {} foo entries", entries.len());
    Ok(())
}

fn parse_foo_enum(src: &str) -> anyhow::Result<Vec<(u16, String)>> {
    // Pattern-match the C++/SQL/lua syntax. Be strict (bail on
    // malformed lines) so silent corruption is loud.
    todo!()
}

2. src/foo.rs — includes generated source

//! Generated table of foo entries scraped from
//! `vendor/server/src/map/enums/foo.h` at build time. Do not edit
//! the entries by hand — modify the scraper or the LSB source.

include!(concat!(env!("OUT_DIR"), "/foo_table.rs"));

pub fn lookup(id: u16) -> Option<&'static str> {
    FOO_TABLE
        .binary_search_by_key(&id, |(k, _)| *k)
        .ok()
        .map(|i| FOO_TABLE[i].1)
}

3. Unit test pinning at least one known entry

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn scraper_extracted_known_entry() {
        // Pick something stable in LSB upstream that's unlikely to
        // be renamed. If this assertion fails, the scraper parsed
        // the source wrong (or LSB renamed a load-bearing entry).
        assert_eq!(lookup(38), Some("SkillGain"));
    }

    #[test]
    fn scraper_extracted_nontrivial_count() {
        // Detects "scraper silently emitted zero rows" bugs.
        assert!(FOO_TABLE.len() > 100);
    }
}

Non-negotiables

1. cargo:rerun-if-changed for every file the scraper reads. Without this, cargo caches stale tables across vendor/ updates — the exact failure mode the scraper exists to prevent.
2. Bail on malformed input. A scraper that silently produces FOO_TABLE: &[] = &[] is worse than no scraper. Use anyhow::bail! on every "expected N fields, got M" branch.
3. Pin at least one entry in a unit test. Catches regressions where the scraper parses but emits garbage.
4. Document the LSB source path at the top of the generated module so a future reader can trace the mirror back.
5. Emit a cargo:warning with the row count so the build log shows the scraper ran and how many entries it produced — this is how the user spots silent regressions in development.

Common LSB source shapes to scrape

Shape	Where in vendor	Parse strategy
C++ enum class	vendor/server/src/map/enums/*.h	Strip braces, match Name = N, lines
SQL INSERT INTO	vendor/server/sql/*.sql	Match INSERT INTO \table` VALUES (...)per line, split on,`
Lua action table	vendor/server/scripts/...	Heavier — only worth it for high-value, stable tables
#define constants	scattered in C++ headers	Match #define NAME VALUE lines
Phoenix-style enum class X : uint8_t	vendor/Phoenix/src/.../*.h	Same as LSB enums; useful when LSB hasn't migrated

When to extend an existing scraper vs add a new one

Extend ffxi-proto/build.rs when the new data is more msg_*-like constants. Add a new build.rs in a different crate when the data has a different consumer or release cadence. Don't put navmesh data and message IDs in the same scraper — keeps cargo:rerun-if-changed narrow so unrelated edits don't trigger rebuilds.

Cross-references

• ffxi-proto/build.rs — multi-enum scraper template
• ffxi-nav/build.rs — single-SQL-file scraper template with load-bearing y/z swap documented inline (see that file's module comment for an example of capturing transformation semantics at scrape time)
• lsb-mirror-check skill — for verifying that the scraped values actually match LSB's runtime usage (scraping the data is necessary but not sufficient)