create-text
Deterministic public-domain text corpus chunker with seeded selectors for reproducible sampling across engineering/government corpora.
معلومات المصدر
- المستودع
- grahama1970/agent-stack-public
- آخر نشاط في المصدر
- ٢٤ سبتمبر ٢٠٢٦ في ١٥:٥١
- لغة SKILL.md المكتشفة
- الإنجليزية
- النجوم
- ٠
- التفرعات
- ٠
خيارات التثبيت
يُحدَّد Prompt الذي يراجع المصدر أولًا بشكل افتراضي. يمكنك التبديل إلى أمر مباشر أو تنزيل نسخة محلية.
مراجعة ملفات المصدر
اقرأ SKILL.md وأي ملفات مرافقة يعرضها SkillsMP قبل أن تقرر التثبيت.
مستكشف الملفات
5 ملفاتعرض SKILL.md
SKILL.md
تعليمات المصدر · معاينة للقراءة فقط- name
- create-text
- description
- Deterministic public-domain text corpus chunker with seeded selectors for reproducible sampling across engineering/government corpora.
- allowed-tools
- Bash, Read, Write
- triggers
- ["create text","text chunker","deterministic text","build text bank","select text chunks"]
- metadata
- {"short-description":"Deterministic corpus text chunker"}
- provides
- ["create-text"]
- composes
- ["ingest-doc","extract-pdf","best-practices-python","agentic-evals"]
- complies
- ["best-practices-skills","best-practices-python"]
- disciplines
- ["data-engineering"]
# /create-text
Deterministic public-domain text corpus chunker with seeded selector.
## Quick Start
```bash
# Build text banks from extracted datalake corpus
./run.sh build
# Build from Wikipedia (specific categories)
./run.sh build --source wikipedia --categories "Cybersecurity" "Avionics"
# Get 16 requirement-style text chunks from defense domain, seed 42
./run.sh select --content-type requirement --domain defense --count 16 --seed 42
# Get prose paragraphs from any domain
./run.sh select --content-type prose --count 8 --seed 7
# List available domains and content types
./run.sh list
```
## Content Types
| Type | Description | Source |
|------|-------------|--------|
| `prose` | Continuous paragraphs (3+ sentences) | PDF corpus, Wikipedia |
| `bullet_list` | Enumerated/bulleted items | PDF corpus |
| `requirement` | Unconditional SHALL/MUST clauses | PDF corpus (MIL-STD, ECSS, NASA-STD) |
| `conditional_requirement` | When/If/Unless + SHALL guard clauses | PDF corpus (engineering specs) |
| `compound_requirement` | SHALL + enumerated sub-obligations | PDF corpus (rare — extractors split blocks) |
| `heading` | Section headers at various levels | PDF corpus |
| `latex_equation` | Math equations (raw LaTeX or unicode) | PDF corpus (arxiv) |
| `table_cell` | Table content fragments | PDF corpus |
| `glossary` | Definition-style entries | PDF corpus, Wikipedia |
| `footnote` | Footnote/endnote text | PDF corpus |
## Domains
Auto-discovered from datalake at `/mnt/storage12tb/extractor_corpus/`.
Common domains: `arxiv`, `defense`, `government`, `engineering`, `nasa`, `nist`, `ietf`, `wikipedia`.
## Interface
```python
from create_text import create_text
# Deterministic: same seed = same output
chunks = create_text(content_type="requirement", domain="defense", count=16, seed=42)
# Returns: list[dict] with keys: text, content_type, domain, source_doc, block_id
```
## Banks
Built once via `./run.sh build`, stored as JSON at `/mnt/storage12tb/text_banks/`.
One file per domain: `defense.json`, `arxiv.json`, `wikipedia.json`, etc.
## Common Mistakes
```bash
# WRONG: Generate synthetic text with an LLM
# → Use real extracted text for ground truth
# WRONG: Random selection without seed
# → Always pass a seed for reproducibility
# WRONG: Build banks on every call
# → Banks are built once and cached. Use ./run.sh build explicitly.
```
عرض على GitHub