moai-formats-data

name	moai-formats-data
description	Data format specialist covering TOON encoding, JSON/YAML optimization, serialization patterns, and data validation for modern applications. Use when optimizing data for LLM transmission, implementing high-performance serialization, validating data schemas, or converting between data formats.
license	Apache-2.0
compatibility	Designed for Claude Code
allowed-tools	Read Write Edit Grep Glob mcp__context7__resolve-library-id mcp__context7__get-library-docs
user-invocable	false
metadata	{"version":"2.0.0","category":"library","status":"active","updated":"2026-01-11","modularized":"true","tags":"formats, data, toon, serialization, validation, optimization","author":"MoAI-ADK Team"}
triggers	{"keywords":["serialization","data format","json","yaml","toon","validation","schema","optimization"]}

Data Format Specialist

Quick Reference

Advanced Data Format Management - Comprehensive data handling covering TOON encoding, JSON/YAML optimization, serialization patterns, and data validation for performance-critical applications.

Core Capabilities:

• TOON Encoding: 40-60% token reduction vs JSON for LLM communication
• JSON/YAML Optimization: Efficient serialization and parsing patterns
• Data Validation: Schema validation, type checking, error handling
• Format Conversion: Seamless transformation between data formats
• Performance: Optimized data structures and caching strategies
• Schema Management: Dynamic schema generation and evolution

When to Use:

• Optimizing data transmission to LLMs within token budgets
• High-performance serialization/deserialization
• Schema validation and data integrity
• Format conversion and data transformation
• Large dataset processing and optimization

Quick Start:

Create a TOONEncoder instance and call encode with a dictionary containing user and age fields to compress the data. The encoded result achieves 40-60% token reduction. Call decode to restore the original data structure.

Create a JSONOptimizer instance and call serialize_fast with a large dataset to achieve ultra-fast JSON processing.

Create a DataValidator instance and call create_schema with a dictionary defining name as a required string type. Call validate with the data and schema to check validity.

---

Implementation Guide

Core Concepts

TOON (Token-Optimized Object Notation):

• Custom binary-compatible format optimized for LLM token usage
• Type markers: # for numbers, ! for booleans, @ for timestamps, ~ for null
• 40-60% size reduction vs JSON for typical data structures
• Lossless round-trip encoding/decoding

Performance Optimization:

• Ultra-fast JSON processing with orjson achieving 2-5x faster than standard json
• Streaming processing for large datasets using ijson
• Intelligent caching with LRU eviction and memory management
• Schema compression and validation optimization

Data Validation:

• Type-safe validation with custom rules and patterns
• Schema evolution and migration support
• Cross-field validation and dependency checking
• Performance-optimized batch validation

Basic Implementation

TOON Encoding for LLM Optimization:

Create a TOONEncoder instance. Define data with user object containing id, name, active boolean, and created datetime, plus permissions array. Call encode to compress and decode to restore. Compare sizes to verify reduction.