githubcopilot-image-annotations

name	githubcopilot-image-annotations
description	Annotate screenshots, diagrams, and images with callout rectangles, arrows, labels, and color-coded highlights using PIL. Includes rules for animated GIF annotations with timing and pacing.

Image Annotations

Add visual callouts to any image — screenshots, diagrams, architecture docs, demo frames — using PIL/Pillow. Highlights what changed or what to look at, so reviewers don't have to guess.

When to Use This Skill

Use this skill when you need to:

• Highlight a specific area in a screenshot for a PR description
• Annotate before/after images to show what changed
• Add labels and callouts to diagrams or architecture images
• Create annotated frames for animated GIF demos

Prerequisites

pip install Pillow -q

Color Rules

• Red (#E63946) — only for "bad" / "removed" things (e.g., circling a bug being fixed)
• Yellowish-orange (#FF9F1C) — for neutral highlights ("look here", "new feature", etc.)
• Never use red just because it's eye-catching — red = bad/removed

Font

• Use Ink Free (C:/Windows/Fonts/Inkfree.ttf) for a handwritten look on Windows
• On Linux/macOS, fall back to ImageFont.load_default()
• Size 36 for annotations on ~1400px-wide images
• stroke_width=1 with stroke_fill=<same color as fill> — gives body without being too thick
• Do NOT use white stroke — looks like a bad glow effect

Shapes

• Prefer rounded rectangles over circles/ellipses — less pixelation at edges
• draw.rounded_rectangle([x1, y1, x2, y2], radius=14, outline=color, width=5)
• Padding 18px around the target content

Reference Snippet

from PIL import Image, ImageDraw, ImageFont

# Setup
font = ImageFont.truetype('C:/Windows/Fonts/Inkfree.ttf', 36)  # or load_default()
color = '#FF9F1C'  # orange for highlights
stroke = 5
pad = 18

img = Image.open('screenshot.png')
draw = ImageDraw.Draw(img)

# Rounded rect with padding
draw.rounded_rectangle(
    [x1 - pad, y1 - pad, x2 + pad, y2 + pad],
    radius=14, outline=color, width=stroke
)

# Leader line (same thickness as rect)
draw.line([x2 + pad, cy, x2 + pad + 40, cy - 30], fill=color, width=stroke)

# Label — same-color stroke for body, NO white stroke
draw.text(
    (x2 + pad + 45, cy - 60), 'label text',
    fill=color, font=font, stroke_width=1, stroke_fill=color
)

img.save('annotated.png')

Algorithmic Annotation — annotate.py

For images with multiple elements to annotate, use the annotate.py module below. Save it next to your script and import from it. It handles automatic label placement without overlapping.

Quick start

from annotate import annotate_image

result = annotate_image(
    'screenshot.png',
    [
        {'elem': (560, 275, 635, 390), 'label': 'button', 'draw_box': True},
        {'elem': (105, 453, 236, 470), 'label': 'status text'},
    ],
    debug=True,
)
result.save('annotated.png')

• elem: (x1, y1, x2, y2) tight bounding box — must be exact pixel coordinates
• label: text label (supports \n for multi-line)
• draw_box: if True, draws a rounded rectangle around the element. If False (default), draws a V-arrowhead pointing at the element
• debug: shows targeting rectangles and candidate heatmap for placement validation

Coordinate grid helper

Always use grid_image() before annotating an unfamiliar image. Scaled-down previews display images smaller than actual pixel dimensions — the error compounds as you move away from (0,0).

from annotate import grid_image

grid = grid_image('screenshot.png', step=100)
grid.save('grid.png')

Then verify with small crops:

from PIL import Image
img = Image.open('screenshot.png')
crop = img.crop((x1 - 20, y1 - 20, x2 + 20, y2 + 20))
crop.save('verify.png')

Algorithm overview

1. Ring search: candidates between MIN_ARROW (25px) and MAX_ARROW (120px) from element edge
2. Contrast scoring: prefers placements where label text is readable — abs(avg_brightness - 147) - std * 0.3 - dist * 0.02
3. Joint resolution: candidates computed independently, placed greedily (best score first)
4. Hard blocks: labels cannot overlap any other annotation's element or breathing box
5. Proximity penalty: labels within 40px of other placed boxes get a score penalty
6. Arrow crossing penalty: -50 for arrows crossing already-placed arrows

Debug mode colors

Color	Meaning
Cyan	Target element box (elem + padding)
Gray	Exclusion zone (MIN_ARROW buffer)
Red→Green	Candidate heatmap (red=bad, green=good)
Magenta	Chosen label position
Orange	Final rendered annotation

Arrow styles

• draw_box=True: rounded rectangle + straight line to label, no arrowhead
• draw_box=False: V-shaped arrowhead with rounded line caps

annotate.py — full module

Save this as annotate.py and import from it:

"""
Algorithmic screenshot annotation with automatic label placement.

pip install Pillow numpy
Optional for diff_images: pip install scipy
"""
import math
import numpy as np
from PIL import Image, ImageDraw, ImageFont

# --- Defaults ---
DEFAULT_FONT = 'C:/Windows/Fonts/Inkfree.ttf'
DEFAULT_FONT_SIZE = 32
DEFAULT_COLOR = '#FF9F1C'
DEFAULT_STROKE = 5
MIN_ARROW = 25
MAX_ARROW = 120
TEXT_PAD = 6
BREATH = 18
CROSSING_PENALTY = 50
PROXIMITY_MARGIN = 40
PROXIMITY_PENALTY = 50


def _rect_intersects(a, b):
    return a[0] < b[2] and a[2] > b[0] and a[1] < b[3] and a[3] > b[1]


def _segments_intersect(p1, p2, p3, p4):
    def cross(o, a, b):
        return (a[0] - o[0]) * (b[1] - o[1]) - (a[1] - o[1]) * (b[0] - o[0])
    d1, d2 = cross(p3, p4, p1), cross(p3, p4, p2)
    d3, d4 = cross(p1, p2, p3), cross(p1, p2, p4)
    return ((d1 >   d2 < )  (d1 <   d2 > ))  \
           ((d3 >   d4 < )  (d3 <   d4 > ))


 ():
    x1, y1, x2, y2 = rect
    dx, dy = tx - cx, ty - cy
    tmin, tmax = , 
     lo, hi, p, d  [(x1, x2, cx, dx), (y1, y2, cy, dy)]:
         (d) < :
            
        t0, t1 = (lo - p) / d, (hi - p) / d
         t0 > t1:
            t0, t1 = t1, t0
        tmin, tmax = (tmin, t0), (tmax, t1)
     (cx + dx * tmax, cy + dy * tmax)


 ():
    dx = (a[] - b[], b[] - a[], )
    dy = (a[] - b[], b[] - a[], )
     dx ==   dy == :
         
     math.sqrt(dx** + dy**)


 ():
    cx, cy = (cyan[] + cyan[]) / , (cyan[] + cyan[]) / 
    excl_zone = (cyan[] - MIN_ARROW, cyan[] - MIN_ARROW,
                 cyan[] + MIN_ARROW, cyan[] + MIN_ARROW)
    sx1 = (, cyan[] - MAX_ARROW - pw)
    sy1 = (, cyan[] - MAX_ARROW - ph)
    sx2 = (W - pw, cyan[] + MAX_ARROW)
    sy2 = (H - ph, cyan[] + MAX_ARROW)
    step_x = (, (pw // , MAX_ARROW // ))
    step_y = (, (ph // , MAX_ARROW // ))
    cands = []
     px  (sx1, sx2, step_x):
         py  (sy1, sy2, step_y):
            pink = (px, py, px + pw, py + ph)
             _rect_intersects(pink, excl_zone):
                
            gl, gr = cyan[] - pink[], pink[] - cyan[]
            gt, gb = cyan[] - pink[], pink[] - cyan[]
            hd, vd = (gl, gr, ), (gt, gb, )
            ed = math.sqrt(hd** + vd**)  (hd >   vd > )  (hd, vd)
             ed > MAX_ARROW:
                
            region = pixels[py:py + ph, px:px + pw, :].astype()
            score = (np.mean(region) - ) - np.std(region) * 
            dist = math.sqrt((px + pw/ - cx)** + (py + ph/ - cy)**)
            score -= dist * 
            cands.append(((px, py), score))
     cands


 ():
    placed = []
    all_elem_zones = []
     ann  annots:
        all_elem_zones.append(ann[])
         ann.get(, ):
            c = ann[]
            all_elem_zones.append((c[]-BREATH, c[]-BREATH, c[]+BREATH, c[]+BREATH))
     ann  (annots, key= a: -a[]):
        pw, ph = ann[], ann[]
        cyan = ann[]
        cx, cy = ann[]
        draw_box = ann.get(, )
        best_pos, best_score = , -
        valid = []
         (px, py), score  ann[]:
            pink = (px, py, px + pw, py + ph)
            ok = 
             ez  all_elem_zones:
                 ez == cyan:
                    
                 ann.get(, ):
                    own_viz = (cyan[]-BREATH, cyan[]-BREATH, cyan[]+BREATH, cyan[]+BREATH)
                     ez == own_viz:
                        
                 _rect_intersects(pink, ez):
                    ok = ; 
              ok:
                
             p_pink, p_excl, p_viz, _  placed:
                 _rect_intersects(pink, p_pink)  _rect_intersects(pink, p_excl):
                    ok = ; 
                 p_viz  _rect_intersects(pink, p_viz):
                    ok = ; 
              ok:
                
             p_pink, p_excl, p_viz, _  placed:
                 rect  [p_pink, p_excl, p_viz]:
                     rect  :
                        
                    gap = _rect_gap(pink, rect)
                     gap < PROXIMITY_MARGIN:
                        score -= PROXIMITY_PENALTY * ( - gap / PROXIMITY_MARGIN)
             ez  all_elem_zones:
                 ez == cyan:
                    
                gap = _rect_gap(pink, ez)
                 gap < PROXIMITY_MARGIN:
                    score -= PROXIMITY_PENALTY * ( - gap / PROXIMITY_MARGIN)
            tcx, tcy = px + pw/, py + ph/
            cand_start = _line_rect_exit(tcx, tcy, cx, cy, pink)
             draw_box:
                viz = (cyan[]-BREATH, cyan[]-BREATH, cyan[]+BREATH, cyan[]+BREATH)
                cand_end = _line_rect_exit(cx, cy, tcx, tcy, viz)
            :
                cand_end = _line_rect_exit(cx, cy, tcx, tcy, cyan)
             _, _, _, pa  placed:
                 pa  _segments_intersect(cand_start, cand_end, pa[], pa[]):
                    score -= CROSSING_PENALTY; 
            valid.append(((px, py), score))
             score > best_score:
                best_score, best_pos = score, (px, py)
        ann[] = valid
         best_pos  :
            ann[] = ann[] = ann[] = ann[] = ann[] = 
            
        px, py = best_pos
        pink = (px, py, px + pw, py + ph)
        ann[] = pink
        ann[] = (px + TEXT_PAD, py + TEXT_PAD)
        tcx, tcy = px + pw/, py + ph/
        ann[] = _line_rect_exit(tcx, tcy, cx, cy, pink)
         draw_box:
            viz = (cyan[]-BREATH, cyan[]-BREATH, cyan[]+BREATH, cyan[]+BREATH)
            ann[] = viz
            ann[] = _line_rect_exit(cx, cy, tcx, tcy, viz)
        :
            ann[] = 
            ann[] = _line_rect_exit(cx, cy, tcx, tcy, cyan)
        placed.append((pink, ann[], ann[], (ann[], ann[])))


 ():
    overlay = Image.new(, img.size, (, , , ))
    od = ImageDraw.Draw(overlay)
     ann  annots:
        cands = ann.get(, ann[])
          cands:
            
        pw, ph = ann[], ann[]
        scores = [s  _, s  cands]
        smin, smax = (scores), (scores)
        rng = smax - smin  smax > smin  
         (px, py), score  cands:
            t = (score - smin) / rng
             t < :
                r_c, g_c, b_c = , ( * (t * )), 
            :
                r_c, g_c, b_c = ( * ( - (t-)*)), , 
            alpha_fill = ( +  * t)
            alpha_out = ( +  * t)
            od.rectangle((px, py, px + pw, py + ph),
                         fill=(r_c, g_c, b_c, alpha_fill), outline=(r_c, g_c, b_c, alpha_out), width=)
     ann  annots:
        ez = ann[]
        od.rectangle(ez, fill=(, , , ), outline=(, , , ), width=)
        od.rectangle(ann[], fill=(, , , ), outline=(, , , ), width=)
         ann.get():
            od.rectangle(ann[], fill=(, , , ),
                         outline=(, , , ), width=)
     Image.alpha_composite(img, overlay)


 ():
    draw = ImageDraw.Draw(img)
     ann  annots:
         ann.get():
            draw.rounded_rectangle(ann[], radius=, outline=color, width=stroke_width)
        tpos = ann.get()
        astart, aend = ann.get(), ann.get()
          (tpos  astart  aend):
            
        sx, sy = (astart[]), (astart[])
        ex, ey = (aend[]), (aend[])
        draw.line([(sx, sy), (ex, ey)], fill=color, width=, joint=)
        r = 
        draw.ellipse([(sx-r, sy-r), (sx+r, sy+r)], fill=color)
        draw.ellipse([(ex-r, ey-r), (ex+r, ey+r)], fill=color)
          ann.get(, ):
            angle = math.atan2(ey - sy, ex - sx)
            al, spread = , 
            ax = ex - al * math.cos(angle - spread)
            ay = ey - al * math.sin(angle - spread)
            bx = ex - al * math.cos(angle + spread)
            by = ey - al * math.sin(angle + spread)
            draw.line([((ax), (ay)), (ex, ey)], fill=color, width=)
            draw.line([((bx), (by)), (ex, ey)], fill=color, width=)
             px_, py_  [((ax), (ay)), ((bx), (by))]:
                draw.ellipse([(px_-r, py_-r), (px_+r, py_+r)], fill=color)
        draw.text(tpos, ann[], fill=color, font=font,
                  stroke_width=, stroke_fill=color)
     img


 ():
    
    font = ImageFont.truetype(font_path, font_size)
    img = Image.(image_path).convert()
    pixels = np.array(img)
    W, H = img.size
    annots = []
     i, spec  (annotations):
        eb = spec[]
        em_pad = (, (, (eb[] - eb[]) // ))
        cyan = (eb[] - em_pad, eb[] - em_pad, eb[] + em_pad, eb[] + em_pad)
        lines = spec[].split()
        tw = (font.getbbox(line)[] - font.getbbox(line)[]  line  lines)
        line_h = font.getbbox()[] - font.getbbox()[]
        th = line_h * (lines) +  * ((lines) - )
        pw, ph = tw +  * TEXT_PAD, th +  * TEXT_PAD
        cands = _find_candidates(pixels, W, H, cyan, pw, ph, font)
        annots.append({
            : i,
            : spec[],
            : spec.get(, ),
            : cyan,
            : ((cyan[]+cyan[])/, (cyan[]+cyan[])/),
            : (cyan[]-MIN_ARROW, cyan[]-MIN_ARROW,
                          cyan[]+MIN_ARROW, cyan[]+MIN_ARROW),
            : pw, : ph,
            : cands,
            : ((s  _, s  cands), default=-),
        })
    _resolve_placements(annots, font)
    annots.sort(key= a: a[])
     debug:
        img = _draw_debug(img, annots, color)
    img = _draw_annotations(img, annots, font, color, stroke_width)
     img


 ():
    
     scipy  ndimage
    img_a = Image.(before_path).convert()
    img_b = Image.(after_path).convert()
     img_a.size != img_b.size:
         ValueError()
    arr_a = np.array(img_a, dtype=np.float32)
    arr_b = np.array(img_b, dtype=np.float32)
    W, H = img_a.size
    diff = np.(arr_b - arr_a).(axis=)
    mask = diff > threshold
    dilated = ndimage.binary_dilation(mask, iterations=dilate)
    labeled, n_clusters = ndimage.label(dilated)
    clusters = []
     i  (, n_clusters + ):
        ys, xs = np.where(labeled == i)
         (ys) < min_pixels:
            
        clusters.append(((xs.()), (ys.()),
                          (xs.()), (ys.()), (ys)))
    clusters.sort(key= c: -c[])
    debug_img = 
     debug:
        overlay = img_b.copy().convert()
        norm = np.clip(diff / , , )
        show_mask = diff > 
        r = np.clip((norm * ) * , , ).astype(np.uint8)
        g = np.clip(( - np.(norm - ) * ) * , , ).astype(np.uint8)
        b = np.clip(( - norm) * , , ).astype(np.uint8)
        a = np.where(show_mask, np.clip(norm *  + , , ).astype(np.uint8), )
        heat = Image.fromarray(np.stack([r, g, b, a], axis=), )
        overlay = Image.alpha_composite(overlay, heat)
        draw = ImageDraw.Draw(overlay)
        :
            font = ImageFont.truetype(, )
         OSError:
            font = ImageFont.load_default()
         idx, (x1, y1, x2, y2, px_count)  (clusters):
            draw.rectangle([x1, y1, x2, y2], outline=(, , , ), width=)
            label = 
            bbox = font.getbbox(label)
            tw, th = bbox[] - bbox[], bbox[] - bbox[]
            lx, ly = x1, (, y1 - th - )
            draw.rectangle([lx, ly, lx + tw + , ly + th + ], fill=(, , , ))
            draw.text((lx + , ly + ), label, fill=(, , , ), font=font)
        debug_img = overlay
     clusters, debug_img


 ():
    
    img = Image.(image_path).convert()
    draw = ImageDraw.Draw(img)
    W, H = img.size
    :
        font = ImageFont.truetype(, )
     OSError:
        font = ImageFont.load_default()
     x  (, W, step):
        draw.line([(x, ), (x, H)], fill=(, , , ), width=)
        draw.text((x + , ), (x), fill=(, , , ), font=font)
     y  (, H, step):
        draw.line([(, y), (W, y)], fill=(, , , ), width=)
        draw.text((, y + ), (y), fill=(, , , ), font=font)
     img

Image Diffing

Find what changed between two screenshots programmatically. Use as a safety net for subtle changes — when the difference is obvious, annotate directly instead.

from annotate import diff_images

clusters, debug_img = diff_images(
    'before.png', 'after.png',
    threshold=30,     # pixel difference floor (0-255)
    min_pixels=300,   # ignore tiny noise clusters
    dilate=5,         # merge nearby changed pixels
    debug=True,       # render heatmap overlay
)

# clusters = [(x1, y1, x2, y2, pixel_count), ...] sorted largest-first
if debug_img:
    debug_img.save('diff-debug.png')

# Feed clusters into annotate_image:
annotations = [
    {'elem': (x1, y1, x2, y2), 'label': f'Change #{i+1}', 'draw_box': True}
    for i, (x1, y1, x2, y2, _) in enumerate(clusters[:3])
]

Debug heatmap colors: Blue = small difference, Yellow = medium, Red = large, Cyan boxes = cluster bounding boxes.

When to use: subtle opacity changes, dashed lines, minor color shifts, anti-aliasing differences. When NOT to use: any change you can see by eye — annotate directly for better labels.

Animated GIF Annotations

Different from static images — animations have timing, transitions, and competing visual motion.

Element highlighting

1. Rects for big areas, arrows for small elements — 500x300px area = rect, 200x25px element = arrow
2. Labels go RIGHT NEXT to what they describe — short arrow (30-80px), label adjacent. Viewer's eye shouldn't travel more than ~100px
3. Arrow must not cross its own label — pick the edge closest to the target
4. No bottom bar / subtitle approach — eyes jump between content and bar. Contextual placement only
5. Hero message gets a bigger font — main takeaway 64pt+, detail annotations 38pt

Timing and pacing

6. Fade: 2-frame pop-in at 10fps — 50% → 100% opacity (0.2s total). Easing curves look bad at low FPS
7. Type → pause → annotate — during fast action, show NO annotation. Pause, then add it
8. Variable frame duration — fast during action (100ms), slow during pauses (600-800ms), long hold for hero (500ms)
9. Higher FPS for smooth motion — 10fps minimum for typing/interaction

Pop-in fade implementation

# 2-frame pop-in at 10fps
FADE_ALPHAS = [0.50, 1.00]

for frame_idx in range(total_frames):
    if annotation_just_changed and local_idx < len(FADE_ALPHAS):
        alpha = FADE_ALPHAS[local_idx]
    else:
        alpha = 1.0
    # Apply alpha to annotation elements:
    # - pill background: fill=(r, g, b, int(base_alpha * alpha))
    # - text: fill=(*color, int(255 * alpha))
    # - rect outline: outline=(*color, int(255 * alpha))

Guidelines

1. All elements same thickness — rect width, line width, and visual text weight should feel consistent (~5px)
2. Place labels close to the rect — short leader line (25-35px)
3. Labels can overlap content — the stroke gives enough contrast
4. Show locally first — verify before uploading to a PR
5. Take screenshots at native 1x, control display size in HTML — use <img width="300"> in markdown, never resize with PIL (creates artifacts)
6. Always check Image.open(path).size first — HiDPI screenshots are larger than they appear (150% scaling = 1.5x CSS pixel dimensions)
7. Short labels work better — wide labels have fewer valid placements. Use 1-3 words when possible
8. Verify with debug=True — always check the first annotation of a new image with debug mode

Limitations

• Ink Free font is Windows-only; other platforms need a fallback font
• PIL text rendering is basic — no rich text, no markdown
• Animated GIF annotations require frame-by-frame processing which can be slow for long recordings
• Algorithmic placement works best with 2-6 annotations; more than that may produce crowded results