| name | kinetics-test-time-scaling |
| title | Kinetics: Rethinking Test-Time Scaling Laws |
| version | 0.0.2 |
| engine | skillxiv-v0.0.2-claude-opus-4.6 |
| license | MIT |
| url | https://arxiv.org/abs/2506.05333 |
| keywords | ["scaling-laws","inference-optimization","attention-cost","sparse-attention","reasoning-efficiency"] |
| description | Reveals that memory bandwidth—not computation—dominates test-time costs; proposes eFLOPs metric incorporating both computation and memory, showing 14B+ minimum threshold for reasoning value. |
Kinetics: Memory-Aware Test-Time Scaling
Core Concept
Conventional test-time scaling analysis focuses on computational FLOPs, but actual inference bottlenecks are memory bandwidth constraints. Kinetics introduces eFLOPs (effective FLOPs) that combine computational and memory access costs, revealing fundamental truths: smaller models are less valuable than assumed, a ~14B parameter minimum threshold exists before scaling strategies help, and attention—not parameter count—dominates cost in extended reasoning. Sparse attention emerges as the key complementary approach, enabling 60+ point gains on mathematical reasoning without expensive dense attention.
Architecture Overview
- eFLOPs Metric: Cost model incorporating both computation (FLOPs) and memory bandwidth constraints
- Memory-Aware Analysis: Shows KV cache access costs dominate parameter costs during generation
- Scaling Law Revision: Demonstrates smaller models (<14B) rarely benefit from test-time scaling
- Sparse Attention Paradigm: Reduces quadratic attention cost via block-level top-k sparsity
- Block Top-k Implementation: Practical sparse attention variant achieving 11-26× speedup on H200 GPUs
- Cost Threshold Framework: Guides model selection and scaling strategy for given compute budgets
Implementation
The following code demonstrates eFLOPs cost analysis and sparse attention:
import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import Tuple, List
import numpy as np
class EFLOPsCostModel:
"""
Effective FLOPs cost model incorporating both computation and memory.
"""
def __init__(self, gpu_peak_flops: float = 1.5e15,
memory_bandwidth_gb_s: float = 4.8):
self.peak_flops = gpu_peak_flops
self.bandwidth_bytes_s = memory_bandwidth_gb_s * (1024 ** 3)
def compute_dense_attention_cost(self, seq_len: int, hidden_dim: int,
num_heads: int = 32) -> Tuple[float, float, float]:
"""
Compute computation and memory costs for dense attention.
Returns: (computation_flops, memory_bytes, total_eflops)
"""
compute_flops = 2 * seq_len * seq_len * hidden_dim
q_bytes = seq_len * hidden_dim *
k_bytes = seq_len * hidden_dim *
v_bytes = seq_len * hidden_dim *
attn_matrix_bytes = seq_len * seq_len *
output_bytes = seq_len * hidden_dim *
total_memory_bytes = q_bytes + k_bytes + v_bytes + attn_matrix_bytes + output_bytes
memory_flops = total_memory_bytes
total_eflops = compute_flops + memory_flops
compute_flops, total_memory_bytes, total_eflops
() -> [, , ]:
compute_flops = * seq_len * seq_len * hidden_dim * sparsity_ratio
q_bytes = seq_len * hidden_dim *
k_bytes = seq_len * hidden_dim *
v_bytes = seq_len * hidden_dim *
attn_matrix_bytes = seq_len * seq_len * * sparsity_ratio
output_bytes = seq_len * hidden_dim *
total_memory_bytes = q_bytes + k_bytes + v_bytes + attn_matrix_bytes + output_bytes
memory_flops = total_memory_bytes
total_eflops = compute_flops + memory_flops
compute_flops, total_memory_bytes, total_eflops
() -> :
dense:
_, _, eflops = .compute_dense_attention_cost(seq_len, hidden_dim)
:
_, _, eflops = .compute_sparse_attention_cost(seq_len, hidden_dim)
total_eflops = eflops * num_layers
estimated_latency = total_eflops / .bandwidth_bytes_s
estimated_latency
() -> :
(, ( * (compute_budget_flops / )))
(nn.Module):
():
().__init__()
.hidden_dim = hidden_dim
.num_heads = num_heads
.head_dim = hidden_dim // num_heads
.block_size = block_size
.q_proj = nn.Linear(hidden_dim, hidden_dim)
.k_proj = nn.Linear(hidden_dim, hidden_dim)
.v_proj = nn.Linear(hidden_dim, hidden_dim)
.out_proj = nn.Linear(hidden_dim, hidden_dim)
() -> torch.Tensor:
batch_size, seq_len, _ = x.shape
Q = .q_proj(x).view(batch_size, seq_len, .num_heads, .head_dim)
K = .k_proj(x).view(batch_size, seq_len, .num_heads, .head_dim)
V = .v_proj(x).view(batch_size, seq_len, .num_heads, .head_dim)
Q = Q.transpose(, )
K = K.transpose(, )
V = V.transpose(, )
scores = torch.matmul(Q, K.transpose(-, -)) / (.head_dim ** )
num_blocks = (seq_len + .block_size - ) // .block_size
block_scores = scores.view(batch_size, .num_heads, num_blocks,
.block_size, seq_len)
block_means = block_scores.mean(dim=)
k = (, (seq_len * top_k_ratio))
_, top_block_indices = torch.topk(block_means, k=k, dim=)
sparse_mask = torch.zeros_like(scores)
b (batch_size):
h (.num_heads):
query_block (num_blocks):
query_start = query_block * .block_size
query_end = (query_start + .block_size, seq_len)
i (query_start, query_end):
top_indices = top_block_indices[b, h, query_block]
top_idx top_indices:
sparse_mask[b, h, i, top_idx] =
scores = scores * sparse_mask + ( - sparse_mask) * (-)
attn = F.softmax(scores, dim=-)
output = torch.matmul(attn, V)
output = output.transpose(, ).contiguous()
output = output.view(batch_size, seq_len, .hidden_dim)
output = .out_proj(output)
output
:
():
.cost_model = cost_model
() -> :
threshold = .cost_model.scaling_law_threshold(compute_budget_flops)
threshold
() -> :
dense_cost = .cost_model.compute_dense_attention_cost(seq_len, )[]
sparse_cost = .cost_model.compute_sparse_attention_cost(seq_len, , )[]
(dense_cost - sparse_cost) / dense_cost >
() -> :
sparse_attention model_size < :
log_tokens = np.log(num_reasoning_tokens + )
gains = * ( - np.exp(-log_tokens / ))
gains
Practical Guidance
eFLOPs Computation: Always account for memory bandwidth in cost analysis. Modern GPUs are memory-bound for sequence operations; dense attention on H200 achieves <10% peak FLOP utilization due to memory limitations.
Model Size Threshold: The 14B minimum threshold is empirical but generalizable. Below 14B, test-time scaling provides minimal benefit relative to inference cost. Budget-conscious deployments should use sparse attention instead.
Sparse Attention Tuning: Block size of 64 tokens provides good granularity. Top-k ratio of 0.1 (keeping 10% of attention matrix) is typical; adjust based on accuracy-latency tradeoff.
Sequence Length Scheduling: Sparse attention becomes increasingly valuable as sequences grow. Below 4K tokens, dense may be comparable; above 16K, sparse is strongly preferred.
Benchmarking Strategy: Test both dense and sparse variants on your specific hardware. H200 benefits more from sparse due to higher memory-computation ratio.
Hybrid Approaches: Consider dense attention for early layers (building representations) and sparse for later layers (refinement). This balances accuracy with efficiency.
Reference
Kinetics achieves strong efficiency improvements:
- Block Top-k: 11-26× speedup on H200 GPUs
- Sparse attention gains: 60+ points on MATH-related tasks
- Memory-aware insight: Bandwidth, not computation, is the primary bottleneck
- Scaling threshold: Minimum 14B parameter models for effective test-time scaling
This framework is particularly valuable for reasoning-intensive applications where extended generation is necessary but computational budgets are constrained. The memory-aware perspective corrects widespread misconceptions about scaling benefits for smaller models.
