define-mdp-and-transition-rules

name	Define MDP and Transition Rules
description	This skill should be used when the user asks to "define the MDP", "specify transition dynamics", "set episode termination conditions", "define truncation vs termination", "establish reset conditions", "determine if the problem is model-based or model-free", or "design the Markov Decision Process". Do NOT hallucinate parameters outside the boundaries of Define MDP & Transition Rules.
version	0.1.0

Define MDP & Transition Rules

Define transition dynamics, episodic bounds, and the full Markov Decision Process (MDP) specification. This skill produces a formal breakdown of terminal states, truncation conditions, reset strategy, and the model-based vs model-free targeting decision.

The MDP Tuple

A complete MDP is defined as the tuple $(\mathcal{S}, \mathcal{A}, \mathcal{T}, \mathcal{R}, \gamma)$:

Symbol	Name	Description
$\mathcal{S}$	State Space	All possible states
$\mathcal{A}$	Action Space	All possible actions
$\mathcal{T}(s' \mid s, a)$	Transition Function	Probability of next state given current state and action
$\mathcal{R}(s, a, s')$	Reward Function	Scalar reward signal
$\gamma$	Discount Factor	Future reward discount, typically 0.99

Step 1 — Establish the Horizon and Episode Truncation

Critical distinction — Termination vs. Truncation:

Type	Meaning	Value Bootstrap Rule
Termination	Agent reached a true terminal state (won, died, invalid)	Bootstrap value = 0. No future rewards exist.
Truncation	Time limit hit (max_steps reached), task could continue	Bootstrap value = $V(s_T)$. Agent could have earned more.

This distinction is critical for Value Critic correctness. Mixing termination and truncation signals causes the critic to systematically underestimate value at time boundaries.

In Gymnasium, implement via:

return obs, reward, terminated, truncated, info
# terminated=True → true terminal state
# truncated=True → time limit hit

Define max_steps explicitly. Very few real problems are truly infinite-horizon continuing tasks.

Step 2 — Outline Transition Dynamics

Classify the environment dynamics:

Dynamics Type	Characteristics	Implication
Deterministic	Same action always produces same next state. $\mathcal{T}(s'\mid s,a) \in {0,1}$	Model-Based methods (MuZero, AlphaZero) are applicable.
Stochastic	Noise in transitions. $\mathcal{T}(s'\mid s,a)$ is a distribution	Requires Model-Free methods or stochastic Model-Based methods.

For stochastic environments, specify the noise model:

• Additive Gaussian: s' = f(s, a) + ε, ε ~ N(0, σ²)
• Categorical transition probabilities
• External disturbance bounds

Step 3 — Determine Reset Conditions

Define what "fresh start" means. A poor reset strategy causes the agent to overfit to a single initial trajectory.

Reset randomization strategies:

Strategy	Method	When to Use
Fixed start	Always reset to identical state	Only for deterministic puzzles (e.g. fixed maze)
Uniform random	Sample start state uniformly from valid region	Default for most environments
Curriculum reset	Start near goal early, then expand difficulty	Sparse reward environments
State buffer reset	Reset to states from past successful episodes	Hard exploration problems

Output Format

Provide a formal breakdown with these four sections:

Terminal States (Success/Failure):

• List all conditions that trigger terminated=True
• Separate success terminals from failure terminals

Truncation Condition (Time Limit):

• State max_steps value
• Justify the chosen limit relative to the expected optimal episode length

Reset Randomization Strategy:

• Describe the initial state distribution
• Specify which state variables are randomized and their ranges

Algorithm Targeting:

• Model-Free or Model-Based, with justification based on dynamics classification above

Additional Resources

Reference Files

• references/mdp-formalism.md — Full MDP mathematical formalism, POMDP extensions, and discount factor selection guide

Scripts

• scripts/validate_mdp_spec.py — Validates an MDP specification dict for completeness and consistency