pythermodb-reference-maker

name

pythermodb-reference-maker

description

Extract thermodynamic tables and correlations from references (CSV, PDF, images, or text) and convert them into the project's structured pyThermoDB YAML schema. Supports data tables, constants tables, matrix-parameter tables, and equation-based correlations (e.g., Cp, vapor pressure, density, enthalpy of vaporization), including coefficient parsing and transformation into executable project notation (parms, args, res). Trigger when working with thermodynamic tables, coefficients, correlations, constants, matrix parameters, or YAML formatting. Ensures unit-consistent, schema-compliant, and solver-ready thermodynamic definitions for downstream modeling tools.

Thermodynamic Table to YAML Extractor

Purpose

Convert thermodynamic tables into the structured YAML format used by pyThermoDB references.

Support:

• Data tables such as general component properties.
• Constants tables such as custom constants or reaction constants.
• Matrix-parameter tables such as binary NRTL parameters.
• Equation tables such as Cp, vapor pressure, density, and enthalpy of vaporization.
• Multi-equation systems.
• Optional integral and derivative expressions.
• Optional variable range columns such as Tmin, Tmax, Pmin, or Pmax.

Step 0: Interpret the task

Before processing:

1. Identify input type:
   • CSV
   • PDF/image
   • raw text
   • existing YAML
   • online source or database page
   • article, paper, report, or handbook excerpt
2. Identify table type:
   • data table: component or mixture scalar properties
   • constants table: named constants and scalar/list/dict constant values
   • matrix table: pairwise or matrix-parameter data
   • equation table: coefficients plus formula
3. Identify container format:
   • full reference: REFERENCES -> reference id -> DATABOOK-ID -> TABLES
   • direct table snippet: one or more table names at the top level
4. Identify complexity:
   • single equation
   • multiple equations
   • mixed table patterns
5. Check for:
   • scaled coefficients
   • variable range columns such as Tmin/Tmax
   • duplicate component records in VALUES
   • missing metadata columns
   • special-case rows

Step 0a: Resolve source policy

Before extracting values, determine whether the user supplied or constrained the source.

Source modes:

• user-supplied source: a file, image, pasted text, URL, DOI, article, report, or database page
• user-constrained source: the user says to use only a specific source, such as NIST, DIPPR, Perry's, a named article, or an attached file
• open source search: the user asks for thermodynamic data without naming the source

Rules:

• If the user constrains the source, use only that source unless it is incomplete or internally inconsistent. If another source is needed to resolve a missing formula, unit, equation form, or coefficient definition, ask or clearly label the additional source as supplemental.
• If the user supplies an article, report, PDF, image, or excerpt, extract from that material first. Preserve the article's equation form, coefficient definitions, units, ranges, and stated validity limits before translating into executable pyThermoDB notation.
• If the user asks for NIST or another online thermodynamic database, retrieve the relevant table or page when browsing is available. Record the database name, page/table title, component names, property, equation form, units, and access date in notes or the table description when useful.
• If the user does not name a source, prefer authoritative thermodynamic references and databases. Use exact source attribution in notes so coefficients are traceable.
• Do not blend coefficients from multiple sources into one row unless the output explicitly includes source-disambiguating columns or notes. Report conflicts instead of silently choosing one value.
• Do not invent missing coefficients, ranges, or equation identifiers. Mark missing values as unresolved in notes and produce only the YAML that can be supported by the source.
• Respect source terms and access limits. Summarize copyrighted source text and extract only the factual coefficients, equations, units, and metadata needed for the YAML.

Step 1: Classify table type

Data table

A data table contains component or mixture scalar properties and no evaluable correlation equation.

table-name:
  TABLE-ID: <int>
  DESCRIPTION:
    <text>
  DATA: []
  STRUCTURE:
    COLUMNS: [...]
    SYMBOL: [...]
    UNIT: [...]
    CONVERSION: [...]
  VALUES:
    - [...]

Equation table

An equation table contains coefficients tied to one or more equations.

table-name:
  TABLE-ID: <int>
  DESCRIPTION:
    <text>
  EQUATIONS:
    EQ-1:
      BODY:
        - ...
      BODY-INTEGRAL:
        None
      BODY-FIRST-DERIVATIVE:
        None
      BODY-SECOND-DERIVATIVE:
        None
  STRUCTURE:
    COLUMNS: [...]
    SYMBOL: [...]
    UNIT: [...]
  VALUES:
    - [...]

Do not include DATA or CONVERSION for equation tables.

Equation bodies may contain multiple executable steps before assigning to res[...], such as Tr, tau, expo, and a final converted result. Keep intermediate variables when the source formula is complex or when the project examples use that style.

Equation tables may include variable range columns. Store them as normal columns in STRUCTURE.COLUMNS, STRUCTURE.SYMBOL, STRUCTURE.UNIT, and every VALUES row. Do not put range limits inside EQUATIONS.

Constants table

A constants table contains named constants rather than component rows or evaluable correlations.

table-name:
  TABLE-ID: <int>
  DESCRIPTION:
    <text>
  CONSTANTS: []
  STRUCTURE:
    COLUMNS: [No.,Name,Symbol,State,Value,Unit,Description]
  VALUES:
    - [...]

The Value column may be a scalar, string, dictionary, list, or None.

Matrix table

A matrix table defines pairwise mixture symbols, such as NRTL a, b, c, d, alpha, dg, or tau. Use this table type for activity-model binary interaction parameters and other directional i,j properties.

table-name:
  TABLE-ID: <int>
  DESCRIPTION:
    <text>
  MATRIX-SYMBOL:
    - a constant: a
    - b constant: b
  STRUCTURE:
    COLUMNS: [...]
    SYMBOL: [...]
    UNIT: [...]
  VALUES:
    - [...]

The preferred mixture table shape is item rows with these metadata columns: No., Mixture, Name, Formula, and State.

Rules:

• MATRIX-SYMBOL must be a list. Each entry may be a plain symbol (a) or a one-key mapping (a constant: a). Prefer mappings when a clear description is available because the builder stores labels as description-to-symbol pairs.
• Mixture stores the binary pair identity with the configured delimiter, normally component-1|component-2. The builder compares mixture IDs case-insensitively and normalizes member order, so ethanol|methanol and methanol|ethanol match the same binary pair.
• Name, Formula, and State identify the component represented by that row. The discovery API can match rows by Name-State or Formula-State.
• For every base symbol in MATRIX-SYMBOL, include one column per component position in the binary pair: <symbol>_i_1, <symbol>_i_2.
• Multi-component builds for build_mixture_thermodb_from_reference are encoded as binary pair rows, not as one N-component matrix. For [methanol, ethanol, butyl_methyl_ether], write separate binary Mixture IDs such as methanol|ethanol, methanol|butyl-methyl-ether, and ethanol|butyl-methyl-ether, with two rows for each pair. Do not add <symbol>_i_3 columns for this workflow.
• Each binary mixture must have one row for each component in that pair. A binary item table therefore has two rows per Mixture value.
• Matrix cell direction is row component i to column component j. In the row for component 1, a_i_2 stores a_1_2; in the row for component 2, a_i_1 stores a_2_1.
• Populate every matrix cell declared by STRUCTURE. Use 0 only when the source or model definition actually defines the diagonal or missing term as zero. Do not assume diagonal cells are zero for every model.
• COLUMNS, SYMBOL, UNIT, and every VALUES row must have identical length.
• Do not add DATA or CONVERSION to matrix tables.

Step 2: Interpret CSV

When the source is a CSV in the project style:

• row 1 = COLUMNS
• row 2 = SYMBOL
• row 3 = UNIT
• rows 4+ = VALUES

For data tables, also generate CONVERSION.

Step 3: Transform equations

Do not copy equations symbolically. Convert them into executable YAML using the current project notation from examples/reference/str-ref-1.md:

• coefficients -> inline typed parms['name | symbol | unit'] keys
• variables -> inline typed args['name | symbol | unit'] keys
• result -> inline typed res['name | symbol | unit'] keys

Do not add legacy equation metadata blocks:

• no ARGS
• no PARMS
• no RETURNS

The name, symbol, and unit metadata must live inside each parms[...], args[...], and res[...] key string, and the table-level STRUCTURE must still define the row columns, symbols, and units.

Example source:

ln P = C1 + C2/T + C3 ln T + C4 T^C5

Executable YAML body:

EQ-1:
  BODY:
    - parms['C1 | C1 | 1'] = parms['C1 | C1 | 1']/1
    - parms['C2 | C2 | 1'] = parms['C2 | C2 | 1']/1
    - parms['C3 | C3 | 1'] = parms['C3 | C3 | 1']/1
    - parms['C4 | C4 | 1'] = parms['C4 | C4 | 1']/1
    - parms['C5 | C5 | 1'] = parms['C5 | C5 | 1']/1
    - res['vapor-pressure | VaPr | Pa'] = math.exp(parms['C1 | C1 | 1'] + parms['C2 | C2 | 1']/args['temperature | T | K'] + parms['C3 | C3 | 1']*math.log(args['temperature | T | K']) + parms['C4 | C4 | 1']*(args['temperature | T | K']**parms['C5 | C5 | 1']))

Step 4: Preserve coefficient scaling

If a source table shows scaled headers such as a1 x 10^3, a2 x 10^5, a3 x 10^8, or a4 x 10^11, preserve the scale in UNIT, not in SYMBOL.

Correct pattern:

STRUCTURE:
  COLUMNS: [No.,Name,Formula,State,a0,a1,a2,a3,a4,R,Eq]
  SYMBOL: [None,None,None,None,a0,a1,a2,a3,a4,R,Cp_IG]
  UNIT: [None,None,None,None,1,1E3,1E5,1E8,1E11,1,J/mol.K]

Step 5: Keep extended equation blocks

Equation tables may include:

• BODY
• BODY-INTEGRAL
• BODY-FIRST-DERIVATIVE
• BODY-SECOND-DERIVATIVE

If the source or project template includes integral or derivative forms, keep all of them.

Step 5a: Preserve equation variable ranges

If a source gives valid ranges for an equation variable, preserve them in the equation table. The runtime detects ranges from STRUCTURE.SYMBOL by matching equation argument symbols.

For an argument symbol such as T, valid range symbols include:

• Tmin, Tmax, Tlow, Thigh
• T(min), T(max), T(low), T(high)
• T[min], T[max], T[low], T[high]
• T{min}, T{max}, T{low}, T{high}

Prefer Tmin and Tmax for temperature limits. Use the same unit as the variable, such as K for Tmin/Tmax. Related source columns such as P(Tmin) or P(Tmax) may be preserved as ordinary metadata columns, but they are not temperature range bounds.

Step 6: Fixed schema rules

• Every row must have exactly the same number of values as the number of columns.
• Never shift columns.
• In component-style data and equation tables, define only one VALUES row per component record.
• Use 0 for unused coefficients if that is the active project convention.
• Keep mandatory metadata columns when the project requires them: No., Name, Formula, State.
• For State, use default project codes unless the user explicitly requests another format:
  • g = gas
  • l = liquid
  • s = solid
  • aq = aqueous

Component record uniqueness means one row for each component identity in a table, normally Name plus Formula plus State when those columns exist. Different states are different component identities, so Methane/CH4/g and Methane/CH4/l are both valid rows in the same table. Do not create a second row for the same Name/Formula/State identity, such as another methane gas row, to hold extra coefficients, ranges, or alternate metadata. Merge all columns for that component-state identity into the same row. If the source gives conflicting values for the same component-state identity and table, flag the conflict in notes instead of silently adding a duplicate row.

For matrix tables, row identity is normally Mixture plus row component identity. With state-aware matching, use Mixture plus Name/Formula plus State; with state-ignored matching, use Mixture plus Name or Formula. Do not collapse the two rows of a binary pair into one row, and do not split a single row component's matrix values across duplicate rows for the same mixture/component identity. For multi-component mixtures, do not create a row identity containing all members, such as methanol|ethanol|butyl-methyl-ether. Use one row identity per binary pair and row component.

Step 7: Unit and conversion rules

For data tables, use CONVERSION with internal = stored * conversion.

Use:

• 1 for numeric columns already in target internal units
• None for text columns

For equation tables, do not use CONVERSION; handle scaling and unit adjustments inside the equation body.

Step 8: Formula style

Use the compact formula style used by the project's main reference files.

Prefer:

• C2H4
• C7H8
• C3H6O2

Avoid source-display formulas such as CH2=CH2 or C6H5-CH3 unless the user explicitly asks for source-faithful display formulas.

Step 9: Use bundled resources

Read or run these when needed:

• references/schema_rules.md: table-type schema rules for data, equation, constants, and matrix tables.
• references/equation_patterns.md: equation transformation patterns and math-module rules.
• references/examples.md: examples for valid data, equation, constants, and matrix tables.
• assets/reference_file_template.yaml: starting template for a full REFERENCES: file with mixed table types.
• assets/data_table_template.yaml: starting template for direct data-table snippets.
• assets/equation_table_template.yaml: starting template for direct equation-table snippets.
• assets/constants_table_template.yaml: starting template for direct constants-table snippets.
• assets/matrix_table_template.yaml: starting template for direct matrix-parameter table snippets.
• scripts/csv_to_structure.py: convert project-style CSV headers into YAML arrays.
• scripts/validate_yaml.py: validate YAML shape. It accepts full REFERENCES: files and direct table snippets.
• scripts/check_reference.py: validate that YAML can be loaded by pyThermoDB itself.
• For mixture matrix tables, also compare against diagrams/reference_mixture_table.md when available; it is the detailed contract for MATRIX-SYMBOL, Mixture, row identity, and NRTL/UNIQUAC examples.

Prefer .venv/Scripts/python.exe on Windows when running pyThermoDB-aware scripts inside this repository.

Example validation flow:

.venv/Scripts/python.exe skills/pythermodb-reference-maker/scripts/validate_yaml.py examples/external-ref/source-ref-1.yml
.venv/Scripts/python.exe skills/pythermodb-reference-maker/scripts/check_reference.py examples/external-ref/source-ref-1.yml CUSTOM-REF-1 Custom-Constants

Validation checklist

Before finalizing:

• table type is correct
• DATA exists for data tables
• CONVERSION exists for data tables
• EQUATIONS exists for equation tables
• CONSTANTS exists for constants tables
• MATRIX-SYMBOL exists for matrix tables
• matrix tables use Mixture, Name, Formula, and State metadata columns when they are intended for build_mixture_thermodb_from_reference
• matrix-table base symbols expand to complete directional columns such as a_i_1, a_i_2, and every declared matrix cell has a value
• row width matches column count
• component-style VALUES contain only one row per component identity and state
• symbols, units, and values align exactly
• scaling is handled correctly
• variable range columns are preserved when present in the source
• formulas are normalized to project style unless instructed otherwise
• state values use g, l, s, or aq unless another format was explicitly requested
• no guessed coefficients are presented as exact

Output format

Return:

1. table type
2. notes on corrections or dependencies
3. final YAML block

Strict rules

• Do not guess coefficients.
• Do not drop columns required by the project schema.
• Do not mix table formats.
• Do not duplicate component-state rows in one component-style data or equation table.
• Do not ignore coefficient scaling shown in the source.
• Do not expand or change State values away from g, l, s, or aq unless explicitly requested.

Definition of done

The task is complete only when:

• YAML matches the project schema
• equations are executable in project notation
• units and scales are represented in the correct layer
• data integrity is preserved