kinetics-test-time-scaling

name	kinetics-test-time-scaling
title	Kinetics: Rethinking Test-Time Scaling Laws
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2506.05333
keywords	["scaling-laws","inference-optimization","attention-cost","sparse-attention","reasoning-efficiency"]
description	Reveals that memory bandwidth—not computation—dominates test-time costs; proposes eFLOPs metric incorporating both computation and memory, showing 14B+ minimum threshold for reasoning value.

Kinetics: Memory-Aware Test-Time Scaling

Core Concept

Conventional test-time scaling analysis focuses on computational FLOPs, but actual inference bottlenecks are memory bandwidth constraints. Kinetics introduces eFLOPs (effective FLOPs) that combine computational and memory access costs, revealing fundamental truths: smaller models are less valuable than assumed, a ~14B parameter minimum threshold exists before scaling strategies help, and attention—not parameter count—dominates cost in extended reasoning. Sparse attention emerges as the key complementary approach, enabling 60+ point gains on mathematical reasoning without expensive dense attention.

Architecture Overview

• eFLOPs Metric: Cost model incorporating both computation (FLOPs) and memory bandwidth constraints
• Memory-Aware Analysis: Shows KV cache access costs dominate parameter costs during generation
• Scaling Law Revision: Demonstrates smaller models (<14B) rarely benefit from test-time scaling
• Sparse Attention Paradigm: Reduces quadratic attention cost via block-level top-k sparsity
• Block Top-k Implementation: Practical sparse attention variant achieving 11-26× speedup on H200 GPUs
• Cost Threshold Framework: Guides model selection and scaling strategy for given compute budgets

Implementation

The following code demonstrates eFLOPs cost analysis and sparse attention:

import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import Tuple, List
import numpy as np

class EFLOPsCostModel:
    """
    Effective FLOPs cost model incorporating both computation and memory.
    """
    def __init__(self, gpu_peak_flops: float = 1.5e15,  # H200
                 memory_bandwidth_gb_s: float = 4.8):    # H200
        self.peak_flops = gpu_peak_flops
        self.bandwidth_bytes_s = memory_bandwidth_gb_s * (1024 ** 3)

    def compute_dense_attention_cost(self, seq_len: int, hidden_dim: int,
                                    num_heads: int = 32) -> Tuple[float, float, float]:
        """
        Compute computation and memory costs for dense attention.

        Returns: (computation_flops, memory_bytes, total_eflops)
        """
        # Attention computation: Q @ K^T -> (seq_len, seq_len) -> @ V
        # FLOPs: 2 * seq_len^2 * hidden_dim + seq_len^2 operations
        compute_flops = 2 * seq_len * seq_len * hidden_dim

        # Memory access: load Q, K, V, store output + intermediate (attention matrix)
        # Each is seq_len * hidden_dim
        q_bytes = seq_len * hidden_dim * 
        k_bytes = seq_len * hidden_dim * 
        v_bytes = seq_len * hidden_dim * 
        attn_matrix_bytes = seq_len * seq_len *   
        output_bytes = seq_len * hidden_dim * 

        total_memory_bytes = q_bytes + k_bytes + v_bytes + attn_matrix_bytes + output_bytes

        
        
        memory_flops = total_memory_bytes

        total_eflops = compute_flops + memory_flops

         compute_flops, total_memory_bytes, total_eflops

     () -> [, , ]:
        
        
        
        compute_flops =  * seq_len * seq_len * hidden_dim * sparsity_ratio

        
        q_bytes = seq_len * hidden_dim * 
        k_bytes = seq_len * hidden_dim * 
        v_bytes = seq_len * hidden_dim * 
        attn_matrix_bytes = seq_len * seq_len *  * sparsity_ratio
        output_bytes = seq_len * hidden_dim * 

        total_memory_bytes = q_bytes + k_bytes + v_bytes + attn_matrix_bytes + output_bytes

        memory_flops = total_memory_bytes
        total_eflops = compute_flops + memory_flops

         compute_flops, total_memory_bytes, total_eflops

     () -> :
        
         dense:
            _, _, eflops = .compute_dense_attention_cost(seq_len, hidden_dim)
        :
            _, _, eflops = .compute_sparse_attention_cost(seq_len, hidden_dim)

        
        total_eflops = eflops * num_layers

        
        
        estimated_latency = total_eflops / .bandwidth_bytes_s

         estimated_latency

     () -> :
        
        
        
         (, ( * (compute_budget_flops / )))


 (nn.Module):
    
     ():
        ().__init__()
        .hidden_dim = hidden_dim
        .num_heads = num_heads
        .head_dim = hidden_dim // num_heads
        .block_size = block_size

        .q_proj = nn.Linear(hidden_dim, hidden_dim)
        .k_proj = nn.Linear(hidden_dim, hidden_dim)
        .v_proj = nn.Linear(hidden_dim, hidden_dim)
        .out_proj = nn.Linear(hidden_dim, hidden_dim)

     () -> torch.Tensor:
        
        batch_size, seq_len, _ = x.shape

        
        Q = .q_proj(x).view(batch_size, seq_len, .num_heads, .head_dim)
        K = .k_proj(x).view(batch_size, seq_len, .num_heads, .head_dim)
        V = .v_proj(x).view(batch_size, seq_len, .num_heads, .head_dim)

        Q = Q.transpose(, )  
        K = K.transpose(, )
        V = V.transpose(, )

        
        scores = torch.matmul(Q, K.transpose(-, -)) / (.head_dim ** )

        
        
        num_blocks = (seq_len + .block_size - ) // .block_size
        block_scores = scores.view(batch_size, .num_heads, num_blocks,
                                   .block_size, seq_len)

        
        block_means = block_scores.mean(dim=)  

        
        k = (, (seq_len * top_k_ratio))
        _, top_block_indices = torch.topk(block_means, k=k, dim=)

        
        sparse_mask = torch.zeros_like(scores)
         b  (batch_size):
             h  (.num_heads):
                 query_block  (num_blocks):
                    query_start = query_block * .block_size
                    query_end = (query_start + .block_size, seq_len)

                     i  (query_start, query_end):
                        top_indices = top_block_indices[b, h, query_block]
                         top_idx  top_indices:
                            sparse_mask[b, h, i, top_idx] = 

        
        scores = scores * sparse_mask + ( - sparse_mask) * (-)

        
        attn = F.softmax(scores, dim=-)
        output = torch.matmul(attn, V)  

        
        output = output.transpose(, ).contiguous()
        output = output.view(batch_size, seq_len, .hidden_dim)
        output = .out_proj(output)

         output


 :
    
     ():
        .cost_model = cost_model

     () -> :
        
        threshold = .cost_model.scaling_law_threshold(compute_budget_flops)
         threshold

     () -> :
        
        
        
        

        dense_cost = .cost_model.compute_dense_attention_cost(seq_len, )[]
        sparse_cost = .cost_model.compute_sparse_attention_cost(seq_len, , )[]

        
         (dense_cost - sparse_cost) / dense_cost > 

     () -> :
        
        
          sparse_attention  model_size < :
             

        
        log_tokens = np.log(num_reasoning_tokens + )
        gains =  * ( - np.exp(-log_tokens / ))  

         gains

Practical Guidance

eFLOPs Computation: Always account for memory bandwidth in cost analysis. Modern GPUs are memory-bound for sequence operations; dense attention on H200 achieves <10% peak FLOP utilization due to memory limitations.

Model Size Threshold: The 14B minimum threshold is empirical but generalizable. Below 14B, test-time scaling provides minimal benefit relative to inference cost. Budget-conscious deployments should use sparse attention instead.

Sparse Attention Tuning: Block size of 64 tokens provides good granularity. Top-k ratio of 0.1 (keeping 10% of attention matrix) is typical; adjust based on accuracy-latency tradeoff.

Sequence Length Scheduling: Sparse attention becomes increasingly valuable as sequences grow. Below 4K tokens, dense may be comparable; above 16K, sparse is strongly preferred.

Benchmarking Strategy: Test both dense and sparse variants on your specific hardware. H200 benefits more from sparse due to higher memory-computation ratio.

Hybrid Approaches: Consider dense attention for early layers (building representations) and sparse for later layers (refinement). This balances accuracy with efficiency.

Reference

Kinetics achieves strong efficiency improvements:

• Block Top-k: 11-26× speedup on H200 GPUs
• Sparse attention gains: 60+ points on MATH-related tasks
• Memory-aware insight: Bandwidth, not computation, is the primary bottleneck
• Scaling threshold: Minimum 14B parameter models for effective test-time scaling

This framework is particularly valuable for reasoning-intensive applications where extended generation is necessary but computational budgets are constrained. The memory-aware perspective corrects widespread misconceptions about scaling benefits for smaller models.