create-text
Deterministic public-domain text corpus chunker with seeded selectors for reproducible sampling across engineering/government corpora.
소스 정보
- 저장소
- grahama1970/agent-stack-public
- 최근 소스 활동
- 2026년 9월 24일 15:51
- 감지된 SKILL.md 언어
- 영어
- 스타
- 0
- 포크
- 0
설치 방법
기본적으로 소스를 먼저 확인하는 Prompt가 선택됩니다. 직접 명령으로 전환하거나 로컬 사본을 다운로드할 수도 있습니다.
소스 파일 검토
설치 여부를 결정하기 전에 SKILL.md와 SkillsMP에 표시된 보조 파일을 읽어 보세요.
파일 탐색기
5 개 파일SKILL.md 표시 중
SKILL.md
소스 지침 · 읽기 전용 미리보기- name
- create-text
- description
- Deterministic public-domain text corpus chunker with seeded selectors for reproducible sampling across engineering/government corpora.
- allowed-tools
- Bash, Read, Write
- triggers
- ["create text","text chunker","deterministic text","build text bank","select text chunks"]
- metadata
- {"short-description":"Deterministic corpus text chunker"}
- provides
- ["create-text"]
- composes
- ["ingest-doc","extract-pdf","best-practices-python","agentic-evals"]
- complies
- ["best-practices-skills","best-practices-python"]
- disciplines
- ["data-engineering"]
# /create-text
Deterministic public-domain text corpus chunker with seeded selector.
## Quick Start
```bash
# Build text banks from extracted datalake corpus
./run.sh build
# Build from Wikipedia (specific categories)
./run.sh build --source wikipedia --categories "Cybersecurity" "Avionics"
# Get 16 requirement-style text chunks from defense domain, seed 42
./run.sh select --content-type requirement --domain defense --count 16 --seed 42
# Get prose paragraphs from any domain
./run.sh select --content-type prose --count 8 --seed 7
# List available domains and content types
./run.sh list
```
## Content Types
| Type | Description | Source |
|------|-------------|--------|
| `prose` | Continuous paragraphs (3+ sentences) | PDF corpus, Wikipedia |
| `bullet_list` | Enumerated/bulleted items | PDF corpus |
| `requirement` | Unconditional SHALL/MUST clauses | PDF corpus (MIL-STD, ECSS, NASA-STD) |
| `conditional_requirement` | When/If/Unless + SHALL guard clauses | PDF corpus (engineering specs) |
| `compound_requirement` | SHALL + enumerated sub-obligations | PDF corpus (rare — extractors split blocks) |
| `heading` | Section headers at various levels | PDF corpus |
| `latex_equation` | Math equations (raw LaTeX or unicode) | PDF corpus (arxiv) |
| `table_cell` | Table content fragments | PDF corpus |
| `glossary` | Definition-style entries | PDF corpus, Wikipedia |
| `footnote` | Footnote/endnote text | PDF corpus |
## Domains
Auto-discovered from datalake at `/mnt/storage12tb/extractor_corpus/`.
Common domains: `arxiv`, `defense`, `government`, `engineering`, `nasa`, `nist`, `ietf`, `wikipedia`.
## Interface
```python
from create_text import create_text
# Deterministic: same seed = same output
chunks = create_text(content_type="requirement", domain="defense", count=16, seed=42)
# Returns: list[dict] with keys: text, content_type, domain, source_doc, block_id
```
## Banks
Built once via `./run.sh build`, stored as JSON at `/mnt/storage12tb/text_banks/`.
One file per domain: `defense.json`, `arxiv.json`, `wikipedia.json`, etc.
## Common Mistakes
```bash
# WRONG: Generate synthetic text with an LLM
# → Use real extracted text for ground truth
# WRONG: Random selection without seed
# → Always pass a seed for reproducibility
# WRONG: Build banks on every call
# → Banks are built once and cached. Use ./run.sh build explicitly.
```
GitHub에서 보기