| name | chunk-text-extraction |
| description | Deterministically slice a mainframe or IBM i (AS400) source file (COBOL, CBLLE, SQLCBLLE, CL, CLP, CLLE, JCL, PROC, copybook, BMS) into one raw-text file per chunk. USE FOR: extract CL or COBOL chunks as txt and prepare source text for section-doc-writer. DO NOT USE FOR: chunking, facts extraction, facts slicing, copybook expansion, or markdown generation. |
| argument-hint | Path to a source file (e.g. repos/sample1/COBOL/DEMO100.CBL). Optional --chunk-id <id> for one chunk, --output-dir to override, --manifest to override, --prune to drop stale .txt files, --source-root override. |
Chunk Text Extraction
Deterministic phase-B sibling. Reads a source file + its chunk-manifest.json and emits one raw-text file per chunk under docs/_shared/<basename-with-ext>/chunks/<chunk_id_sanitized>.txt. No LLM calls — pure byte/line slicing, sha-gated against stale manifests.
The .txt files are the only raw-source view a downstream per-section subagent (section-doc-writer, qa-reviewer) needs. They sit beside facts-slices/ so a phase-C agent for chunk X reads exactly two files: its slice JSON and its .txt.
When to Use
- The user asks to "extract chunks as txt", "save each chunk to a file", "dump chunk source", or "give me the source for chunk_id X".
- Phase A has just (re-)written a
chunk-manifest.json and the matching chunks/ directory is missing or stale (different source sha256).
- A single chunk needs to be re-extracted on demand (
--chunk-id <id>).
- Preparing per-section inputs before launching phase-C subagents.
Do not use this skill for chunking (cobol-chunking), facts extraction (cobol-facts-extraction), per-chunk facts slicing (facts-slicing), markdown writing (phase C), or copybook expansion.
Output Contract
One UTF-8 text file per chunk at docs/_shared/<basename-with-ext>/chunks/<chunk_id_sanitized>.txt.
<chunk_id_sanitized> = chunk_id.replace("/", "__") — the default transformation. Same rule as facts-slicing.- Duplicate-chunk-id disambiguation. Phase A should guarantee unique chunk ids; some real-world manifests violate that (e.g. two
data-division/FILLER records at different line ranges). On collision, the second+ occurrence is suffixed with __L<start_line> (and _<n> if even that collides) and a DUPLICATE_CHUNK_ID entry is recorded in errors[]. The authoritative chunk-id → filename map is always index.json::chunks[].file — downstream agents MUST read that, not re-derive the filename from the chunk id.
- Body = the raw source bytes for the inclusive 1-based line range
[chunk.start_line .. chunk.end_line] from the manifest, decoded with the manifest's encoding_detected (default utf-8).
- Line endings are normalized to
\n to keep re-runs byte-identical across OSes; the original column content (including significant trailing spaces inside columns 8–72) is preserved verbatim.
- No header, no chunk-id banner, no line-number gutter. The companion slice JSON already carries
chunk.start_line / chunk.end_line.
- Empty chunks (
start_line == end_line + 1 due to upstream parsing edge cases) produce a zero-byte file and a single errors[] entry on stdout, but still write the file.
Alongside the per-chunk files, a single index is emitted at docs/_shared/<basename-with-ext>/chunks/index.json:
{
"index_version": "1.0",
"source_path": "repos/sample1/COBOL/DEMO100.CBL",
"source_sha256": "<64-hex>",
"manifest_sha256": "<64-hex of the chunk-manifest.json bytes>",
"encoding": "utf-8",
"chunks": [
{
"id": "DEMO100/procedure-division/MAIN-PROCESSING/A000-MAIN",
"file": "DEMO100__procedure-division__MAIN-PROCESSING__A000-MAIN.txt",
"start_line": 22,
"end_line": 26,
"line_count": 5,
"byte_count": 187,
"sha256": "<64-hex of the emitted .txt bytes>"
}
chunks[].sha256 is the integrity gate downstream consumers use to detect a stale .txt without re-reading every file.
Procedure
- Resolve inputs. Locate the source file (same rules as upstream skills: relative to repo root first, then to
source_root from config/sources.yaml). Default manifest is at docs/_shared/<basename-with-ext>/chunk-manifest.json. Reject if either is missing.
- Stale-input check. Hash the source bytes on disk and compare with
chunk-manifest.json::sha256. On mismatch: exit code 4, error on stderr, no .txt written. Re-run cobol-chunking first.
- Run the extractor. Invoke scripts/extract-chunks.py:
python .github/skills/chunk-text-extraction/scripts/extract-chunks.py `
--source repos/sample1/COBOL/DEMO100.CBL
Flags:
--chunk-id <id> — extract ONLY that chunk (must exist in the manifest); default is all chunks.--output-dir <path> — override docs/_shared/<basename-with-ext>/chunks.--source-root <root> — override sources.yaml::source_root.--manifest <path> — override the default chunk-manifest location.--prune — delete any pre-existing .txt in the output directory whose chunk id is NOT in the current manifest (default: keep stale files and warn).--no-index — skip writing index.json (useful when extracting a single chunk on demand).
- Report. The script prints
OK: wrote N chunks to <dir> plus total .txt written / skipped / pruned, the source/manifest sha256 prefixes, and any entries from errors[]. Do not echo the chunk bodies.
Batch / Parallel Extraction
When the user asks to extract chunks for more than one file, do not loop the python script inline. Use scripts/extract_chunks_batch.ps1:
pwsh -File .github/skills/chunk-text-extraction/scripts/extract_chunks_batch.ps1 `
-Paths repos/sample1/COBOL/*.CBL
Parameters (mirrors cobol-facts-extraction / facts-slicing):
-Files, -Paths, -Manifest <json|csv> — input selection.-SourceRoot, -OutputDir — per-run overrides.-Prune — pass --prune to every child invocation.-Throttle <int> — defaults to [Environment]::ProcessorCount.-ResultsXml <path> — defaults to chunk_text_results.xml.-ExcludeExtensions <string[]> — case-insensitive extensions skipped when expanding -Paths. Default: .cob, .inp, .itt (copybooks and JCL inline-data are consumed by reference upstream, not chunked standalone).
The runner is deterministic (sorted inputs/results, no timestamps), captures per-file stdout/stderr/exit, persists results via Export-Clixml, and exits 1 if any child failed.
Ad-hoc helpers go in temp/. Same convention as the other phase-B skills.
Report only the summary line (DONE total=… ok=… failed=…) plus any failures, matching the single-file reporting contract in step 4.
Determinism Rules
- No LLM calls.
- Re-running on an unchanged source + manifest pair MUST yield byte-identical
.txt files AND a byte-identical index.json (sorted chunks[], stable key order, no timestamps).
- Line endings always normalized to
\n in the output — the input may be CRLF, LF, or mixed; the emitted bytes are LF-only.
- Filename sanitization is
chunk_id.replace("/", "__") by default. The only non-deterministic-looking case is duplicate chunk ids (a phase-A bug), where the second+ occurrence gets a __L<start_line> suffix — still deterministic given the same manifest. Chunk ids that exceed the OS path limit fail loudly with code=CHUNK_ID_TOO_LONG.
- Stale
source sha256 (manifest vs. on-disk source) is a hard error, not a warning. Extracting against a manifest whose line numbers no longer match the source would silently mis-bind every chunk.
--prune is opt-in. Default behavior is additive (write/overwrite per-chunk files; never delete) so a partial run does not destroy a known-good directory.
Resources
Pipeline Wiring
config/pipeline-dag.yaml phase B lists skills/chunk-text-extraction/scripts/extract-chunks.py as a sibling of cobol-facts-extraction's extract-facts.py. Both run per source after phase A, both consume the same chunk-manifest.json, and neither depends on the other — they can be parallelized freely. Outputs land at docs/_shared/<file>/chunks/<chunk_id>.txt (with / → __ filename sanitization).
Common Pitfalls
- Stale manifest. If the source was edited after
cobol-chunking ran, the line numbers in the manifest no longer point at the right ranges. Step 2 catches this. Always re-run phase A first.
- Encoding mismatch. If the manifest claims
utf-8 but the source is actually EBCDIC or CP1252, the extractor will still write bytes — they just may not be human-readable. Trust the manifest's encoding_detected and fix it in phase A if wrong.
- CRLF on Windows checkouts. The extractor reads via universal-newline mode and writes LF, so the output is stable regardless of git's
core.autocrlf setting. Do not introduce a --preserve-eol flag unless a downstream agent explicitly needs it.
- Filename collisions. Phase A is supposed to guarantee unique chunk ids, but some legacy manifests have duplicates. This skill disambiguates by suffixing
__L<start_line> and emits a DUPLICATE_CHUNK_ID error entry. Always look up filenames via index.json::chunks[].file, never by re-sanitizing the chunk id, or you will mis-bind to the wrong .txt for duplicated ids.
--chunk-id without --no-index. Extracting one chunk without --no-index will overwrite index.json with a single-entry file. Pass --no-index for surgical single-chunk re-extraction.
