data-preprocessing

Data preprocessing is a critical step in machine learning pipelines that transforms raw data into a format suitable for model training. This skill covers data cleaning, feature engineering, normalizat

Ir a la instalación

Repositorio: diegosouzapw/awesome-omni-skill
Última actividad en el origen: 28 de febrero de 2026 a las 04:29
Idioma detectado de SKILL.md: inglés
Estrellas: 50
Forks: 19

Opciones de instalación

De forma predeterminada está seleccionado el prompt que primero revisa el origen. Puedes cambiar a un comando directo o descargar una copia local.

Revisa los archivos de origen

Lee SKILL.md y los archivos complementarios que muestra SkillsMP antes de decidir si quieres instalarlo.

Volver a la búsqueda Volver a la instalación

Data Preprocessing

Skill Profile

(Select at least one profile to enable specific modules)

Overview

Data preprocessing is a critical step in machine learning pipelines that transforms raw data into a format suitable for model training. This skill covers data cleaning, feature engineering, normalization, encoding categorical variables, scaling, augmentation, pipeline creation, and preprocessing for different data types.

Why This Matters

Data Quality: Ensures clean, consistent data
Model Performance: Proper preprocessing improves model accuracy
Feature Quality: Engineering creates more informative features
Training Stability: Scaling ensures stable training
Reproducibility: Pipelines ensure consistent preprocessing

Core Concepts & Rules

1. Core Principles

Follow established patterns and conventions
Maintain consistency across codebase
Document decisions and trade-offs

2. Implementation Guidelines

Start with the simplest viable solution
Iterate based on feedback and requirements
Test thoroughly before deployment

Inputs / Outputs / Contracts

Skill Composition

Depends on: None
Compatible with: None
Conflicts with: None
Related Skills: None

Quick Start / Implementation Example

Review requirements and constraints
Set up development environment
Implement core functionality following patterns
Write tests for critical paths
Run tests and fix issues
Document any deviations or decisions

# Example implementation following best practices
def example_function():
    # Your implementation here
    pass

Assumptions

Data is in a format that can be processed
Sufficient data for imputation and encoding
Domain knowledge for feature engineering

Compatibility

Python: Full support
Pandas: Full support
NumPy: Full support
Scikit-learn: Full support
PyTorch: Partial support

Test Scenario Matrix (QA Strategy)

Type	Focus Area	Required Scenarios / Mocks
Unit	Core Logic	Must cover primary logic and at least 3 edge/error cases. Target minimum 80% coverage
Integration	DB / API	All external API calls or database connections must be mocked during unit tests
E2E	User Journey	Critical user flows to test
Performance	Latency / Load	Benchmark requirements
Security	Vuln / Auth	SAST/DAST or dependency audit
Frontend	UX / A11y	Accessibility checklist (WCAG), Performance Budget (Lighthouse score)

Technical Guardrails & Security Threat Model

1. Security & Privacy (Threat Model)

Top Threats: Injection attacks, authentication bypass, data exposure

Data Handling: Sanitize all user inputs to prevent Injection attacks. Never log raw PII
Secrets Management: No hardcoded API keys. Use Env Vars/Secrets Manager
Authorization: Validate user permissions before state changes

2. Performance & Resources

Execution Efficiency: Consider time complexity for algorithms
Memory Management: Use streams/pagination for large data
Resource Cleanup: Close DB connections/file handlers in finally blocks

3. Architecture & Scalability

Design Pattern: Follow SOLID principles, use Dependency Injection
Modularity: Decouple logic from UI/Frameworks

4. Observability & Reliability

Logging Standards: Structured JSON, include trace IDs request_id
Metrics: Track error_rate, latency, queue_depth
Error Handling: Standardized error codes, no bare except
Observability Artifacts:
- Log Fields: timestamp, level, message, request_id
- Metrics: request_count, error_count, response_time
- Dashboards/Alerts: High Error Rate > 5%

Agent Directives & Error Recovery

(ข้อกำหนดสำหรับ AI Agent ในการคิดและแก้ปัญหาเมื่อเกิดข้อผิดพลาด)

Thinking Process: Analyze root cause before fixing. Do not brute-force.
Fallback Strategy: Stop after 3 failed test attempts. Output root cause and ask for human intervention/clarification.
Self-Review: Check against Guardrails & Anti-patterns before finalizing.
Output Constraints: Output ONLY the modified code block. Do not explain unless asked.

Definition of Done (DoD) Checklist

Tests passed + coverage met
Lint/Typecheck passed
Logging/Metrics/Trace implemented
Security checks passed
Documentation/Changelog updated
Accessibility/Performance requirements met (if frontend)

Anti-patterns / Pitfalls

⛔ Don't: Log PII, catch-all exception, N+1 queries
⚠️ Watch out for: Common symptoms and quick fixes
💡 Instead: Use proper error handling, pagination, and logging

Reference Links & Examples

Internal documentation and examples
Official documentation and best practices
Community resources and discussions

Versioning & Changelog

Version: 1.0.0
Changelog:
- 2026-02-22: Initial version with complete template structure