brain-datafield-exploration-general

name	brain-datafield-exploration-general
description	Provides 6 proven methods to evaluate new datasets on the WorldQuant BRAIN platform. Includes methods for checking coverage, non-zero values, update frequency, bounds, central tendency, and distribution. Use when the user wants to understand a specific datafield (e.g., "what is this field?", "how often does it update?").

6 Ways to Evaluate a New Dataset

This skill provides 6 methods to quickly evaluate a new datafield on the WorldQuant BRAIN platform. For the complete guide and detailed examples, see reference.md.

Important: Run these simulations with Neutralization: None, Decay: 0, Test Period: P0Y0M. Metrics: Check Long Count and Short Count in the IS Summary.

1. Basic Coverage Analysis

• Expression: datafield (or vec_op(datafield) for vectors)
• Insight: % Coverage (Long Count + Short Count) / Universe Size.

2. Non-Zero Value Coverage

• Expression: datafield != 0 ? 1 : 0
• Insight: Real coverage (excluding zeros). Distinguishes missing data (NaN) from actual zero values.

3. Data Update Frequency Analysis

• Expression: ts_std_dev(datafield, N) != 0 ? 1 : 0
• Insight: Frequency of updates. Vary N:
  • N=5 (Week): Low count implies weekly updates.
  • N=22 (Month): Monthly updates.
  • N=66 (Quarter): Quarterly updates.

4. Data Bounds Analysis

• Expression: abs(datafield) > X
• Insight: Check value range. Vary X (e.g., 1, 10, 100) to check scale (e.g., is it normalized -1 to 1?).

5. Central Tendency Analysis

• Expression: ts_median(datafield, 1000) > X
• Insight: Typical values over time (5-year median). Vary X to find the center.

6. Data Distribution Analysis

• Expression: X < scale_down(datafield) && scale_down(datafield) < Y
• Insight: Distribution shape. scale_down maps to 0-1. Vary X and Y (e.g., 0.1-0.2) to check buckets.

Note on Vector Data

If the datafield is a VECTOR type, wrap it in a vector operator first (e.g., vec_sum(datafield) or vec_mean(datafield)).