| name | Define State Space |
| description | This skill should be used when the user asks to "define the state space", "design the observation space", "specify what the agent observes", "determine the agent's inputs", "assess the Markov property", "design observation normalization", or "formulate the observation vector". Do NOT hallucinate parameters outside the boundaries of Define State Space ($S$). |
| version | 0.1.0 |
Define State Space ($S$)
Formulate and design the optimal State Space representation for an RL problem. This skill produces a formal StateSpace specification including space type, dimensions, Markov assessment, normalization requirements, and a physical observation index mapping.
Step 1 — Identify the Core Variables
Enumerate the absolute minimum set of physical/mathematical values required to uniquely describe the current state. Group by variable type:
| Type | Examples |
|---|
| Positions | x, y, z coordinates, joint angles |
| Velocities | Linear velocity, angular velocity, joint velocities |
| Relative metrics | distance_to_target, angle_to_goal (preferred over absolute coordinates) |
| Categorical / binary | Inventory flags, weapon loaded, door open |
| Temporal | Steps remaining, cooldown timer |
| External | Opponent positions, wind speed |
Prefer relative over absolute coordinates. Instead of (x_agent, y_agent) and (x_target, y_target), pass (rel_dx, rel_dy). This removes global position dependence and forces the agent to learn a translation-invariant, generalizable policy.
Step 2 — Assess the Markov Property
The Markov Property requires: $P(s_{t+1} \mid s_t, a_t) = P(s_{t+1} \mid s_1, ..., s_t, a_1, ..., a_t)$
The current observation must be sufficient to predict the next state. Test each variable:
| Scenario | Problem | Solution |
|---|
| Only position available, but velocity needed | Agent cannot infer trajectory direction | Add velocity variables OR use frame-stacking |
| Partially occluded environment | Agent cannot observe all relevant entities | POMDP treatment: use RNN/LSTM policy |
| Hidden opponent intentions | True state partially unobservable | POMDP treatment: use memory-augmented policy |
If the observation is not Markovian, formally classify the problem as a POMDP (Partially Observable MDP). This mandates using an LSTM/GRU policy or frame-stacking in the downstream algorithm design.
Step 3 — Dimensionality Reduction and Feature Engineering
Before finalizing the observation vector, apply these reduction checks:
- Remove absolute global frames → replace with relative frames.
- Remove redundant derivable quantities → if
distance = sqrt(dx² + dy²) is computable from dx, dy already in obs, omit it.
- Collapse symmetric representations → if two joints are always symmetric, represent their difference rather than both values.
- Cap or log-scale unbounded quantities → wealth, cumulative distance, time-alive grow unboundedly and should be log-transformed or normalized with running statistics.
Step 4 — Normalization and Bounds Strategy
Every observation element must have a defined normalization strategy before passing to a neural network:
| Bound Type | Normalization Strategy | Tool |
|---|
Hard bounded [a, b] | Scale to [-1.0, 1.0] statically: (x - a) / (b - a) * 2 - 1 | Manual rescaling in env |
| Soft bounded (approx range known) | Scale by known typical range | Manual rescaling |
| Unbounded (wealth, distance) | Running mean/std normalization | VecNormalize wrapper or RunningMeanStd |
Image pixels [0, 255] | Divide by 255.0 → [0, 1] | Observation wrapper |
Flag all unbounded elements explicitly — these require a VecNormalize or RunningMeanStd wrapper in the training pipeline.
Output Format
When utilizing this skill, generate:
StateSpace:
Type: [Box / Discrete / MultiBinary / Dict]
Dimensions: [Numeric size of flat observation vector]
Markovian: [True / False]
POMDP_Treatment: [None / FrameStack(N) / LSTM / GRU]
Normalizations_Required: [List of wrappers needed]
Follow with a detailed observation index table:
| Index | Variable Name | Physical Meaning | Raw Range | Normalized Range |
|---|
| 0 | rel_dx | Relative X distance to target | [-50m, +50m] | [-1.0, 1.0] |
| 1 | rel_dy | Relative Y distance to target | [-50m, +50m] | [-1.0, 1.0] |
| 2 | vel_x | Agent X velocity | [-10, +10 m/s] | [-1.0, 1.0] |
Additional Resources
Reference Files
references/observation-design-patterns.md — Frame stacking, POMDP treatment, Dict observation spaces, image observation preprocessing
Scripts
scripts/validate_state_space.py — Validates a state space specification dict and checks normalization coverage for all dimensions
