samtok-mask-tokenization

name	samtok-mask-tokenization
title	SAMTok: Representing Any Mask with Two Words
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2601.16093
keywords	["mask-tokenization","multimodal-models","segmentation","discrete-representation","pixel-tasks"]
description	Tokenize any region mask into two special tokens, enabling pixel-wise tasks like segmentation and region understanding in multimodal language models without architectural changes. Use when adapting vision-language models to perform pixel-level understanding and manipulation.

SAMTok: Efficient Mask Tokenization

This skill demonstrates how to represent arbitrary region masks with just two special tokens, enabling vision-language models to perform pixel-wise tasks like segmentation and spatial understanding through standard language prediction.

When to Use

• Adding segmentation capability to existing vision-language models
• Building models for region-level visual understanding
• Interactive image editing and manipulation with language models
• Pixel-level tasks without modifying model architecture
• Systems needing both global image and local region understanding

When NOT to Use

• Simple image classification (doesn't need masks)
• Extremely high-resolution pixel-perfect tasks (two tokens may be limited)
• Tasks requiring dense pixel-level predictions (continuous masks better)
• Systems where architectural modifications aren't a constraint

Key Concept

SAMTok enables language models to reason about image regions by converting arbitrary masks into just two special tokens. Instead of representing masks as:

• Dense pixel-level features (expensive)
• Continuous coordinate regression (hard for LMs)
• Complex polygon representations (non-standard)

SAMTok produces: <REGION_START> ... <REGION_END> where the content between encodes the mask in a format LMs understand.

This integrates seamlessly with existing language model prediction—segmentation becomes token prediction.

Implementation Pattern

Implement mask tokenization for language models:

# Pseudocode for SAMTok mask tokenization
class SAMTok:
    def __init__(self, vocab_size, num_mask_tokens=2):
        self.vocab_size = vocab_size
        self.region_start_token = vocab_size + 1
        self.region_end_token = vocab_size + 2

    def mask_to_tokens(self, mask):
        # Convert binary mask to compact token representation
        
        

        
        compressed = .compress_mask(mask)

        
        tokens = []
        tokens.append(.region_start_token)

        
         chunk  compressed:
            token = .encode_chunk(chunk)
            tokens.append(token)

        tokens.append(.region_end_token)

         tokens

     ():
        
        

        boundary_points = .extract_contour(mask)

        
        quantized = .quantize_points(boundary_points, grid_size=)

         quantized

     ():
        
        
         ((chunk)) % (.vocab_size - )

     ():
        
        
        

        mask_tokens = tokens[tokens.index(.region_start_token) +  :
                            tokens.index(.region_end_token)]

        
        boundary_points = []
         token  mask_tokens:
            chunk = .decode_token(token)
            boundary_points.extend(chunk)

        
        mask = .reconstruct_from_boundary(boundary_points, target_shape)

         mask

     ():
        
        
        
        

        mask_tokens = .mask_to_tokens(region_mask)

        
        full_sequence = image_tokens + mask_tokens + [output_token]

         full_sequence