| name | supervisor-protocol |
| description | Use when coordinating across pipeline modules — tracing config ownership (databases.yaml, project.yaml, ontology_mappings.yaml), diagnosing silent failures, or integrating a new data source end-to-end. Covers the six contracts that fail silently (source name consistency, TSV stems, column agreement, node-before-relationship ordering, OWL name validity, credential injection) and the seven-step new-source checklist. |
Pipeline Data Flow
databases.yaml + .env
↓
BaseParser (download + parse) → data/raw/<source>/
↓
DataFrame dict → export_tsv → data/processed/<source>/<name>.tsv
↓
OntologyPopulator (ontology_mappings.yaml + project.yaml) → data/output/ontology_populated.rdf
↓
MemgraphExporter → data/output/nodes_*.csv, edges_*.csv, import.cypher
Config File Ownership
| File | Owns |
|---|
config/databases.yaml | Which parsers run; constructor args and credentials |
config/project.yaml | Published OWL name table (active node/edge types); disease scope; ontology paths |
config/ontology_mappings.yaml | TSV column → OWL property mappings; node/relationship entry order |
Changing a databases.yaml key breaks: data/processed/ subdirectory, ontology_mappings.yaml prefix, and PARSERS dict in src/main.py — all must be updated together.
Cross-Module Contracts
These six rules must hold for the pipeline to produce correct output. Violations fail silently.
1. Source name consistency — databases.yaml key = PARSERS key (main.py) = ontology_mappings.yaml entry prefix = data/processed/<source>/ subdirectory name. All four must be identical strings. Note: BaseParser.source_name is derived from the class name (ClassName.replace('Parser','').lower()), which controls data/raw/<classname>/ and may differ from the databases.yaml key — this is an expected split (e.g., MEDLINECooccurrenceParser → raw dir medlinecooccurrence, processed dir medline_cooccurrence).
2. TSV filename stems — Keys in parse_data() return dict become TSV filename stems (e.g., key "gene_disease" → gene_disease.tsv). Each source_filename in ontology_mappings.yaml must exactly match one of these stems.
3. Column name agreement — Every column name referenced in ontology_mappings.yaml (iri_column_name, subject_column_name, object_column_name, data_property_map keys, merge_column.source_column_name, filter_column) must appear as a column in the corresponding TSV. get_schema() must exactly match what parse_data() produces.
4. Node-before-relationship ordering — In ontology_mappings.yaml, all node entries must precede all relationship entries. The populate step resolves relationships by matching against already-loaded individuals; a relationship entry processed before its subject or object type exists produces zero edges with no error.
5. OWL name validity — node_type and relationship_type values in ontology_mappings.yaml must be OWL classes/properties that exist in data/ontology/ontology.rdf AND appear as active (uncommented) entries in project.yaml node_types/edge_types. Data property names (in data_property_map, *_match_property, merge_column) are not in project.yaml — verify them against the ontology or existing mapping entries for the same class.
6. _env credential injection — _resolve_env_vars() in main.py strips _env suffix and replaces value with os.environ.get(VAR). If the env var is unset, the value becomes None and a WARNING is logged — no hard error. The parser constructor must declare the stripped parameter name. Only add _env keys when the env var is guaranteed present.
Known Silent Failures
| Symptom | Root cause |
|---|
| Zero edges for a source | Relationship entry precedes node entry in ontology_mappings.yaml |
| Populate loads 0 rows | Source name mismatch across configs (contract 1) |
| TSV column not found at populate | Column name mismatch between parser and mappings (contract 3) |
| Credential silently None | _env key added but env var not set in .env |
| OWL property not resolved | Name in mappings doesn't match active entry in project.yaml |
| Stale graph output | MemgraphExporter silently overwrites without warning |
validate_config() on OntologyPopulator checks contracts 3 and 5 partially but does NOT check merge_column.data_property. It is never called automatically — must be run manually before a full pipeline run.
New Source Integration Checklist
Complete all steps in order before running the full pipeline:
- Create parser class extending
BaseParser in src/parsers/<source>_parser.py — implement download_data(), parse_data(), get_schema()
- Add import and add to
__all__ in src/parsers/__init__.py
- Register in
PARSERS in src/main.py: "<databases_yaml_key>": ClassName
- Add entry to
config/databases.yaml with enabled: true and any required args
- Add credentials to
.env if the parser uses _env keys
- Add
ontology_mappings.yaml entries — node entries first, then relationship entries
- Activate any new OWL class or property names in
config/project.yaml node_types / edge_types
After step 4, run python src/main.py --source <key> to produce TSVs.
References
