Skip to main content
great-expectations-validator Data quality validation skill using Great Expectations for schema validation, expectation suites, data documentation, and automated data quality checks in ML pipelines.
Aller à l'installation Skills Marketplace Découvrez et explorez les compétences IA créées par la communauté.
Installer avec Codex ou Claude Copiez ce prompt, collez-le dans Codex, Claude ou un autre assistant, puis laissez-le vérifier la page du skill et l'installer pour vous.
Afficher les détails du prompt Une commande directe contourne le prompt de vérification. Examinez la source avant de l'exécuter.
npx skills add https://github.com/a5c-ai/babysitter --skill great-expectations-validatorLa commande reste sur une seule ligne. Faites défiler horizontalement pour la vérifier avant de la copier.
Vous préférez une copie locale ? Téléchargez les fichiers actuellement disponibles dans SkillsMP.
Télécharger Zip Téléchargement... Métiers associés SOC
Basé sur la classification professionnelle SOC
Reference for querying the Atlas knowledge graph through its MCP tools — the SECONDARY enrichment/comparison layer that adds best-practice context to systems you have ALREADY scanned from your real sources (`az`, repos, dirs). Use when you need to look up nodes, edges, kinds, clusters, stats, or wiki pages in Atlas to compare against your real inventory. (atlas graph, query atlas, atlas mcp, search the graph, graph neighbors, atlas record, atlas kinds, enrichment layer)
Atlas turns your STATED NEED into a real systems atlas by SCANNING your actual sources (Azure via `az`, git repos, local dirs) and process/data mining them, THEN enriching against the Atlas knowledge graph. Use this skill when asked to inventory/map your real systems, scan your cloud + repos + directories, mine the real processes or data they contain, or collect their real constraints/gotchas. (atlas, scan my systems, inventory our azure account, map my repos, real systems atlas, process mining, data mining, collect nuances, system discovery)
assimilate-popular-workflows This skill should be used when the user asks to "find skills in the wild", "assimilate popular workflows", "discover SKILL.md files in repos", "research external skills", "find workflow patterns", "survey the skill landscape", "what skills exist out there", or wants to investigate public repositories for extractable processes, babysitter plugins, and reusable procedural insights. Searches GitHub for SKILL.md files, classifies repos by archetype, and maintains structured research under docs/reference-repos/.
Explorateur de fichiers
2 fichiers name great-expectations-validator description Data quality validation skill using Great Expectations for schema validation, expectation suites, data documentation, and automated data quality checks in ML pipelines. allowed-tools Read, Grep, Write, Bash, Edit, Glob graph {"domains":["domain:data-science"],"specializations":["specialization:data-science-ml"],"skillAreas":["skill-area:data-quality","skill-area:data-pipeline-testing"],"roles":["role:data-scientist","role:ml-ops-engineer"],"workflows":["workflow:data-quality-monitoring"]}
Great Expectations Validator
Validate data quality using Great Expectations for comprehensive data testing, documentation, and quality monitoring.
Overview
This skill provides capabilities for data quality validation using Great Expectations (GX), the leading open-source library for data quality. It enables creation and execution of expectation suites, data documentation generation, and integration with ML pipelines.
Capabilities
Expectation Suite Management
Create and configure expectation suites
Define expectations for columns and tables
Validate data against expectations
Store and version expectation suites
Data Validation
Schema validation (column presence, types)
Statistical validation (distributions, ranges)
Referential integrity checks
Custom SQL-based expectations
Regex pattern matching
Data Documentation
Generate data documentation (Data Docs)
Create profiling reports
Document validation results
Build data dictionaries
Pipeline Integration
Checkpoint configuration and execution
Batch request management
Action-based workflows (notifications, storage)
Integration with Airflow, Prefect, Dagster
Custom Expectations
Define domain-specific expectations
Parameterized expectations
Multi-column expectations
Row-condition based expectations
Prerequisites
Installation pip install great_expectations>=0.18.0
Optional Connectors
pip install great_expectations[sqlalchemy]
pip install great_expectations[s3]
pip install great_expectations[gcs]
pip install great_expectations[azure]
pip install great_expectations[spark]
Usage Patterns
Initialize Great Expectations Project
Create Expectation Suite from Profiler import great_expectations as gx
context = gx.get_context()
datasource = context.sources.add_pandas("my_datasource" )
data_asset = datasource.add_csv_asset("customers" , filepath_or_buffer="customers.csv" )
batch_request = data_asset.build_batch_request()
expectation_suite = context.add_or_update_expectation_suite("customer_suite" )
validator = context.get_validator(
batch_request=batch_request,
expectation_suite_name="customer_suite"
)
validator.expect_column_to_exist("customer_id" )
validator.expect_column_values_to_be_unique("customer_id" )
validator.expect_column_values_to_not_be_null("customer_id" )
validator.expect_column_values_to_be_between("age" , min_value=0 , max_value=120 )
validator.expect_column_values_to_be_in_set("status" , ["active" , "inactive" , "pending" ])
validator.expect_column_values_to_match_regex("email" , r"^[\w\.-]+@[\w\.-]+\.\w+$" )
validator.save_expectation_suite(discard_failed_expectations=False )
Validate Data with Checkpoint import great_expectations as gx
context = gx.get_context()
checkpoint = context.add_or_update_checkpoint(
name="customer_checkpoint" ,
validations=[
{
"batch_request" : {
"datasource_name" : "my_datasource" ,
"data_asset_name" : "customers"
},
"expectation_suite_name" : "customer_suite"
}
],
action_list=[
{
"name" : "store_validation_result" ,
"action" : {"class_name" : "StoreValidationResultAction" }
},
{
"name" : "update_data_docs" ,
"action" : {"class_name" : "UpdateDataDocsAction" }
}
]
)
result = checkpoint.run()
if result.success:
print ("Validation passed!" )
else :
print ("Validation failed!" )
for validation_result in result.run_results.values():
for result in validation_result.results:
if not result.success:
print (f"Failed: {result.expectation_config.expectation_type} " )
Common Expectations
validator.expect_column_to_exist("column_name" )
validator.expect_column_values_to_be_of_type("column_name" , "int64" )
validator.expect_table_column_count_to_equal(10 )
validator.expect_column_values_to_not_be_null("column_name" )
validator.expect_column_values_to_be_null("deprecated_column" )
validator.expect_column_values_to_be_unique("id_column" )
validator.expect_compound_columns_to_be_unique(["col1" , "col2" ])
validator.expect_column_values_to_be_between("age" , min_value=0 , max_value=120 )
validator.expect_column_min_to_be_between("score" , min_value=0 )
validator.expect_column_max_to_be_between("score" , max_value=100 )
validator.expect_column_values_to_be_in_set("status" , ["A" , "B" , "C" ])
validator.expect_column_distinct_values_to_be_in_set("category" , ["cat1" , "cat2" ])
validator.expect_column_values_to_match_regex("email" , r"^[\w\.-]+@[\w\.-]+\.\w+$" )
validator.expect_column_value_lengths_to_be_between("code" , min_value=5 , max_value=10 )
validator.expect_column_mean_to_be_between("value" , min_value=50 , max_value=100 )
validator.expect_column_stdev_to_be_between("value" , min_value=0 , max_value=20 )
validator.expect_column_proportion_of_unique_values_to_be_between("id" , min_value=0.9 )
Integration with Babysitter SDK
Task Definition Example const dataValidationTask = defineTask ({
name : 'great-expectations-validation' ,
description : 'Validate data quality using Great Expectations' ,
inputs : {
dataPath : { type : 'string' , required : true },
expectationSuiteName : { type : 'string' , required : true },
checkpointName : { type : 'string' },
failOnError : { type : 'boolean' , default : true }
},
outputs : {
success : { type : 'boolean' },
validationResults : { type : 'object' },
failedExpectations : { type : 'array' },
dataDocsUrl : { type : 'string' }
},
async run (inputs, taskCtx ) {
return {
kind : 'skill' ,
title : `Validate data: ${inputs.expectationSuiteName} ` ,
skill : {
name : 'great-expectations-validator' ,
context : {
operation : 'validate' ,
dataPath : inputs.dataPath ,
expectationSuiteName : inputs.expectationSuiteName ,
checkpointName : inputs.checkpointName ,
failOnError : inputs.failOnError
}
},
io : {
inputJsonPath : `tasks/${taskCtx.effectId} /input.json` ,
outputJsonPath : `tasks/${taskCtx.effectId} /result.json`
}
};
}
});
MCP Server Integration
Using gx-mcp-server {
"mcpServers" : {
"great-expectations" : {
"command" : "uvx" ,
"args" : [ "gx-mcp-server" ] ,
"env" : {
"GX_CONTEXT_ROOT" : "./great_expectations"
}
}
}
}
Available MCP Tools
gx_list_datasources - List configured datasourcesgx_list_expectation_suites - List expectation suitesgx_run_checkpoint - Execute a checkpointgx_validate_data - Validate data against suitegx_get_validation_results - Retrieve validation results
ML Pipeline Integration
Training Data Validation def validate_training_data (df, suite_name="training_data_suite" ):
"""Validate training data before model training."""
context = gx.get_context()
datasource = context.sources.add_pandas("training_data" )
data_asset = datasource.add_dataframe_asset("df" )
batch_request = data_asset.build_batch_request(dataframe=df)
checkpoint = context.add_or_update_checkpoint(
name="training_validation" ,
validations=[{
"batch_request" : batch_request,
"expectation_suite_name" : suite_name
}]
)
result = checkpoint.run()
if not result.success:
failed = [r for r in result.run_results.values()
for r in r.results if not r.success]
raise ValueError(f"Training data validation failed: {len (failed)} expectations failed" )
return True
Feature Quality Checks
validator.expect_column_values_to_not_be_null("feature_1" , mostly=0.95 )
validator.expect_column_values_to_be_between("feature_1" , min_value=-3 , max_value=3 )
validator.expect_column_proportion_of_unique_values_to_be_between("categorical_feature" , min_value=0.001 )
validator.expect_column_kl_divergence_to_be_less_than("feature_1" ,
partition_object=reference_distribution,
threshold=0.1 )
Best Practices
Version Expectation Suites : Store suites in version controlUse Checkpoints : Always validate through checkpoints for consistencySet Mostly Parameter : Allow for small data quality issues with mostly=0.95Generate Data Docs : Document your data for team visibilityFail Fast : Validate data early in pipelinesCustom Expectations : Create domain-specific expectations for your use case
References 