| name | stylus-input |
| description | Reference for implementing per-platform pen/stylus input backends behind the `stylus-junk` crate's `StylusAdapter` seam. Cross-platform strategy, decision matrix, and links to per-platform details (iOS, Android, Windows, macOS, Linux, Web). |
Stylus input — cross-platform reference
stylus-junk centralizes pen/touch/mouse logic in a single StylusAdapter. Each platform backend translates OS-native events into a small, platform-agnostic shape and calls the adapter. Backends are thin; the adapter owns identity, timestamps, proximity state, estimated/revise cycles, and focus-loss cancellation.
This skill is the launchpad. Jump to a platform file for implementation detail.
- macOS — done. NSEvent local monitor + focus-loss observer. Reference for all other platforms.
- iOS — UITouch + UIEvent coalesced/predicted, UIPencilInteraction, hover via UIHoverGestureRecognizer.
- Windows — Pointer Input API (WM_POINTER) with SetWindowSubclass, GetPointerPenInfoHistory.
- Linux — two paths: Wayland tablet-v2 and X11/XInput2. Both ship.
- Android — MotionEvent via
android-activity, with historical samples and optional MotionPredictor.
- Web — Pointer Events on the canvas, getCoalescedEvents / getPredictedEvents,
touch-action: none.
The adapter seam
Every backend produces two kinds of records and feeds them into the adapter on the main thread:
*RawSample — one physical sample: position (physical px, client-relative), timestamp (monotonic seconds), pressure (0..=1), tilt (x_deg, y_deg, each -90..=90), twist_deg (0..=360 or signed), tangential_pressure (-1..=1), button_mask, device_id, pointing_device_type, source_phase (Down/Move/Up), origin tag.*ProximitySample — per-tool: device_id, unique_id (serial if available), pointing_device_type, ToolCaps bitfield, is_entering.
The macOS path in crates/stylus-junk/src/adapter.rs uses MacTabletRawSample / MacTabletProximitySample and entry points handle_mac_raw / handle_mac_proximity / on_focus_lost. Each new platform should mirror this shape:
- Define
<platform>RawSample and <platform>ProximitySample in the adapter crate (pub(crate)).
- Define
pub(crate) fn handle_<platform>_raw and handle_<platform>_proximity on StylusAdapter, following the same Estimated+Revise pattern already in handle_mac_raw.
- Put OS-specific translation (objc2 / windows-sys / wayland-client / ndk / wasm-bindgen) in a sibling file (
<platform>_tablet.rs). The adapter stays platform-agnostic.
Don't leak OS types into stylus-junk::adapter. The benefit is uniform behavior and testability — the macOS tests drive handle_mac_raw directly without any AppKit dependency; each new platform gets the same treatment.
What the adapter already handles (do not re-implement in backends)
- PointerId allocation per gesture (monotonic counter;
PointerId::MOUSE reserved for mouse).
- Timestamp translation via
translate_mac_timestamp / equivalent — pull this into a shared PlatformTimestampAnchor when you add a new path (first iOS or Windows backend; the extraction is a small refactor).
- Optimistic caps when a first sample arrives before any proximity.
- Estimated + Revise cycle: first Down sample emitted as
SampleClass::Estimated { update_index }, next sample emits a StylusEvent::Revise refining pressure/tilt/twist/tangential.
- Mouse-suppression gate (
active_pen_pointer) so winit's duplicate mouse events during a pen stroke are dropped.
- Focus-loss cancellation (
on_focus_lost) synthesizing Cancel phases for every active pointer.
- Proximity-out-with-active-stroke → synthesizes Cancel.
Every backend calls these existing entry points. What backends must do is translate OS native events and, where the OS provides them, deliver proximity events with the right cap bits and Down samples with the right source_phase. The adapter handles everything else.
Feature matrix
| Feature | macOS | iOS | Windows | Linux (Wl) | Linux (X11) | Android | Web |
|---|
| Pressure | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| Tilt X/Y | ✓ | ✓ (via altitude/azimuth) | ✓ | ✓ | ✓ (libinput) | ✓ (via tilt+orient) | ✓ |
| Twist | ✓ (Wacom Art Pen) | ✓ (Pencil Pro, 17.5+) | ✓ (Surface Slim Pen 2) | ✓ | limited | — | ✓ |
| Tangential | ✓ (airbrush) | — | — | ✓ (slider) | ✓ (Abs Z airbrush) | — | ✓ |
| Hover / proximity | ✓ | ✓ (M1+ iPads, 16.1+) | ✓ (INRANGE) | ✓ (explicit) | patchy | ✓ (stylus HW-dep) | ✓ (synthesized) |
| Eraser tool | ✓ (pointing device type) | via UITouchProperties + hardware | ✓ (PEN_FLAG_ERASER + INVERTED) | ✓ (tool type) | ✓ (separate device) | ✓ (TOOL_TYPE_ERASER) | ✓ (buttons & 0x20) |
| Coalesced / history | winit-native | UIEvent.coalescedTouches | GetPointerPenInfoHistory | per-frame batching | valuator deltas | getHistorical* | getCoalescedEvents |
| Predicted | — | UIEvent.predictedTouches | — | — | — | MotionPredictor (API 33) or Jetpack | getPredictedEvents |
| Stable per-pen ID | uniqueID | — (by update index within gesture) | sourceDevice handle | hardware_serial | synthesized | InputDevice | pointerId (per-gesture only) |
Palm rejection — consistent policy
All platforms: while any pen pointer is in proximity or contact, suppress concurrent touch pointers on the canvas. The adapter's active_pen_pointer gate is the enforcement point for winit-routed events. Backends that surface touch independently (Wayland, XI2, Android MotionEvent) should apply the same rule before handing touch to the engine.
Additionally: honor platform cancel signals.
- iOS:
touchesCancelled.
- Android:
ACTION_CANCEL or MotionEvent.FLAG_CANCELED.
- Windows:
WM_POINTERCAPTURECHANGED.
- Wayland: compositor
proximity_out mid-stroke.
- X11:
XI_ProximityOut mid-stroke (treat as synthesized Up + Cancel).
Decision matrix (what to pick when there's more than one path)
| Platform | Options considered | Choice | Why |
|---|
| iOS | PencilKit vs raw UITouch | raw UITouch + UIEvent | PencilKit owns the surface; we have our own renderer. |
| iOS | Overlay UIView vs swizzle winit's | overlay with hitTest pass-through for non-pencil | Survives winit version bumps; finger touches still reach winit for egui. |
| Windows | WM_POINTER vs RealTimeStylus vs WinRT Ink | WM_POINTER | MS's own recommended modern path; in-thread; covers all axes. |
| Windows | Subclass HWND vs extend winit | SetWindowSubclass | Symmetric to macOS approach; no winit fork. |
| Linux | Wayland tablet vs X11 XI2 vs evdev | both tablet+XI2 | Session-dependent; XWayland means X11 path still matters. Evdev dies under sandboxing. |
| Linux | Shared wl_display vs separate connection | shared (multiple queues) | Tablet events are scoped to surfaces owned by the client; separate connections lose focus routing. |
| Linux/X11 | Master pointer vs slave devices | slave devices | Master aggregates away per-device identity and axis resolution. |
| Android | winit Touch vs raw MotionEvent | raw MotionEvent via android-activity | winit's Touch drops tilt/buttons/historical/tool type. |
| Android | MotionPredictor vs Jetpack MotionEventPredictor | Jetpack | Back-ports prediction to API 24+, same surface. |
| Web | Pointer Events vs WebHID/WebUSB vs Touch Events | Pointer Events | Portable; no permission prompt; covers all axes. |
| Web | winit's pointer events vs own listeners | own listeners on canvas | winit's web backend drops pressure/tilt/coalesced/predicted. |
Implementation order suggestion
Testable-on-hand devices first: Windows → Linux (both paths) → iOS. Then: Web → Android.
- Windows first because the Pointer API is a single well-specified path and Surface/Wacom coverage is broad; tests the generalization of the existing mac-centric adapter code.
- Linux second because it has the highest architectural risk (two paths, winit surgery for Wayland). Doing it early flushes out adapter design issues.
- iOS third because it needs an overlay
UIView subclass via objc2::define_class! and is fiddly but tractable.
- Web and Android last — both are straightforward once the adapter has settled.
Adapter refactors the first non-mac backend should do
The existing adapter has macOS-specific fields (mac_epoch, pending_estimated). When you add the second platform:
- Rename
MacEpoch → PlatformTimestampAnchor, move to adapter.rs top, keep the mac-specific translate_mac_timestamp as a thin wrapper (or generalize).
- Keep
pending_estimated — it's already platform-agnostic.
- Add a
SampleClass::Predicted { predict_index } to aj-core::Sample when implementing the first platform with prediction (iOS, Android, or Web). Predicted samples never commit to the scene; the engine draws them to the Placeholder layer only.
Cross-cutting gotchas
- Physical vs logical pixels. The adapter expects physical pixels, canvas-relative. Each backend is responsible for the conversion. Document which space the backend is in at the top of each file.
- Monotonic timestamps. All OS timestamps we use are monotonic (CLOCK_MONOTONIC-equivalent); none go backwards across sleep-for-our-process-alive-long-enough. Wall-clock time is never used.
- Device serial portability. macOS gives us
uniqueID. Wayland gives us hardware_serial. Windows / Android / iOS / X11 / Web do not give a stable per-pen serial across sessions — use device-id hashes and document the limitation.
- Proximity semantics vary. Wayland and Windows have explicit proximity events. iOS and Android surface proximity only if the device hardware reports hover (Apple M1+ iPad; Pixel Tablet with USI; Samsung S-Pen). X11 is patchy. Web has no explicit proximity but can synthesize from
pointerover+pressure===0. The adapter tolerates proximity being absent; backends should emit it when they can.
- Coexistence with winit mouse events. When the OS fires both pen and synthetic mouse events (macOS, Windows, sometimes Linux drivers), the adapter's
active_pen_pointer gate suppresses the duplicate. Backends don't need to swallow mouse themselves except on Windows where OEM drivers send WM_MOUSEMOVE independently of WM_POINTER (swallow in the subclass).
Testing approach that survives across platforms
The macOS tests in stylus-junk/src/adapter.rs drive handle_mac_raw and handle_mac_proximity directly with pure-Rust fixtures — no AppKit, no device. Every new platform gets the same treatment: handle_<platform>_raw is public-in-crate, and tests feed hand-constructed sample streams.
Golden-image tests (render a pen stroke from a replayed sample log, dssim-compare to committed golden) are the end-to-end check. Record a short stroke from real hardware on each platform we own; replay through the whole stack.
Where to start editing code
crates/stylus-junk/src/lib.rs — module registration and public types (StylusEvent, Phase).crates/stylus-junk/src/adapter.rs — platform-agnostic state machine. Extend with handle_<platform>_* entry points.crates/stylus-junk/src/<platform>_tablet.rs — new, per platform.crates/stylus-junk/Cargo.toml — add [target.'cfg(...)'.dependencies] for each platform's FFI crates.crates/aj-app/src/main.rs — install the backend (pass Rc<RefCell<StylusAdapter>>, RAII guard stored on the App).
The macOS pair (macos_tablet.rs + adapter.rs::handle_mac_*) is the reference implementation. Read it before writing a new backend.
Where the research came from
Each per-platform file has a References section at the bottom pointing to the canonical vendor docs. Summary of best-source-per-topic:
- Apple (iOS / macOS) —
developer.apple.com/documentation/uikit and /appkit are authoritative. WWDC session videos (e.g. WWDC24 10214 for Pencil Pro) are the best source for why APIs look the way they do. objc2-* crates on docs.rs for the Rust bindings.
- Microsoft (Windows) —
learn.microsoft.com/en-us/windows/win32/inputmsg/ for the pointer API. Microsoft's own inking guidance under /windows/apps/design/input/pen-and-stylus-interactions is good for intent; the Win32 API pages are the implementation reference. windows-sys crate on docs.rs for bindings.
- Google (Android) —
developer.android.com/reference/android/view/MotionEvent is canonical. NDK reference at /ndk/reference/group/input covers the C API we actually call. ndk and android-activity on docs.rs.
- Wayland / X11 (Linux) —
wayland.app/protocols/tablet-v2 is the browsable spec; source of truth is the XML in wayland-protocols. XInput2 spec at x.org/releases/current/doc/inputproto/XI2proto.txt. libinput docs at wayland.freedesktop.org/libinput/doc/latest/tablet-support.html for behavior notes. xf86-input-wacom wiki on GitHub for Wacom-on-X11 specifics. wayland-client / wayland-protocols / x11rb on docs.rs.
- W3C / MDN (Web) —
w3c.github.io/pointerevents/ is the spec; MDN at developer.mozilla.org/en-US/docs/Web/API/Pointer_events is the practical reference with compatibility tables. web-sys on docs.rs for bindings.
When reading any of these, verify against the docs.rs version of the crate actually in use — API surfaces (objc2-ui-kit, windows-sys, ndk, wayland-protocols, web-sys) shift between releases. The per-platform files name specific crate versions current at time of writing; treat them as a floor and update on lock-file bump.
