spikeinterface-skill

name	spikeinterface-skill
description	Domain knowledge for building extracellular electrophysiology pipelines with SpikeInterface: loading data with extractors, preprocessing, running spike sorters, post-processing via SortingAnalyzer, quality metrics, curation, comparison, visualization, and export.
domain	electrophysiology
version	1.0.0
authors	["Claude (AI-assisted)"]
review_status	ai-generated

SpikeInterface Skill

1. Purpose

This skill encodes practical knowledge of the SpikeInterface Python framework — a unified API for extracellular electrophysiology spike sorting.

It captures, with verbatim signatures pulled from source, how to:

• Read raw recordings from ~40 acquisition formats and write results back out.
• Chain lazy preprocessors (filter, CMR, phase-shift, motion correction).
• Run internal and external spike sorters (Kilosort family, Mountainsort, SpykingCircus, Tridesclous, Lupin, HerdingSpikes, ...) either natively or inside Docker/Singularity containers.
• Build a SortingAnalyzer and populate it with postprocessing extensions (waveforms, templates, spike amplitudes, unit/spike locations, correlograms, PCA, template similarity).
• Compute quality metrics (misc + PCA-based + spiketrain + template) and use them for automated curation.
• Compare sortings pairwise, against ground truth, or across multiple sorters.
• Visualize with plot_* widgets across matplotlib, ipywidgets, ephyviewer, figpack, and spikeinterface_gui backends (sortingview is deprecated).
• Export to Phy, IBL alignment GUI, Pynapple, or a self-contained HTML report.
• Build custom pipelines out of the low-level sortingcomponents (peak detection, localization, selection, clustering, template matching).
• Generate synthetic ground-truth data and benchmark sorters/components.

2. When to Use This Skill

Trigger this skill when the user's task involves any of the following:

• Loading extracellular ephys files (.bin, .dat, Open Ephys, SpikeGLX, Neuropixels, Blackrock, Plexon, NWB, MEArec, Intan, .h5, .nwb, .zarr, ...).
• Preprocessing multichannel electrode traces (bandpass, notch, CMR, whitening, phase shift, drift/motion correction).
• Running a spike sorter or comparing sorters.
• Constructing a SortingAnalyzer, computing extensions, or asking about analyzer.compute(...).
• Computing quality metrics (SNR, ISI violations, presence ratio, amplitude cutoff, drift, silhouette, nearest-neighbor isolation, ...) or PCA/template metrics.
• Curation — manual (Phy, sortingview) or automated (metrics thresholding, auto_merge, model-based cleaning).
• Ground-truth or cross-sorter comparison.
• Visualization of traces, rasters, waveforms, templates, unit summary, drift, motion, agreement matrices.
• Exporting to Phy / IBL / Pynapple / HTML report.
• Custom sorter development using sortingcomponents (peak detection, localization, clustering, matching).
• Simulating recordings (generate_ground_truth_recording, generate_drifting_recording, hybrid injection) or benchmarking with SorterStudy / BenchmarkStudy.

3. Reference Files (Progressive Disclosure)

References live under references/ and are organized into 12 subdirectories, one per SpikeInterface submodule. Each subdirectory has its own INDEX.md — a short table listing every leaf file, its scope, and when to read it. Start by reading only the INDEX.md for the relevant submodule, then open the specific leaf file(s) it points at. Every leaf file is kept under ~300 lines so you can load only what the current task needs.

Path	Scope	When to open its INDEX
references/core/INDEX.md	Base data-structure classes (BaseExtractor, BaseRecording, BaseSorting, BaseEvent, BaseSnippets, slicers, Numpy extractors), the SortingAnalyzer class + built-in analyzer extensions + ChannelSparsity + Templates, plus the core toolbelt (generate, aggregation/slicing, IO extractors, loading, job_tools, globals, datasets, recording_tools, core_tools, Motion).	You need method signatures on BaseRecording / BaseSorting / SortingAnalyzer, are picking a storage format (memory / binary_folder / zarr), configuring parallelism (n_jobs, chunk_duration), downloading test data, or handling Motion / drift objects.
references/extractors/INDEX.md	Every read_* recording extractor (SpikeGLX, Open Ephys, Blackrock, Intan, Neuralynx, Plexon/Plexon2, MEArec, NWB, IBL streaming, MDA, MCS, TDT, cbin, Zarr, ...), all sorting extractors (read_kilosort, read_phy, read_klusta, read_mda_sorting, read_nwb_sorting, ...), event extractors, snippets extractors, MATLAB / Neuropixels helpers, and registry dicts.	User wants to load a specific raw file format, needs the exact read_<format>(...) signature or required extras (e.g. neo[ced], pynwb, mtscomp), or is reading a sorter's on-disk output / events / snippets.
references/preprocessing/INDEX.md	Lazy preprocessors organized by family: filtering (bandpass_filter, notch_filter, highpass_filter, gaussian_filter, causal_filter), referencing (, ), spatial filtering, bad-channel handling, artifact / silence handling, scaling & normalization, clipping, resampling, dtype conversion, whitening, channel padding, deep-learning denoising, motion correction, plus recipe pipelines.

How to navigate. Every INDEX.md is short (≤ 100 lines) and contains a table of leaf-file | scope | when-to-read. Read the INDEX first, jump to one or two leaves, and only open more if the task requires them. This keeps your context small even for a broad task like "build a full pipeline" — the SKILL body plus 2–3 leaves is usually all you need loaded at once.

4. Pipeline Overview

The canonical SpikeInterface workflow:

Load                Preprocess              Sort                 Analyze                 Post-process           Metrics                Curate                Visualize / Export
----                ----------              ----                 -------                 ------------           -------                ------                -------------------
extractors      ->  preprocessing       ->  sorters          ->  create_              ->  postprocessing    ->  metrics.quality    ->  curation          ->  widgets
read_*(...)         bandpass_filter         run_sorter(          sorting_analyzer(       analyzer.compute(     compute_quality_        auto_merge_units      plot_*(analyzer,...)
                    common_reference        sorter_name,         sorting, recording,     "waveforms",          metrics(analyzer)       apply_sortingview_
                    phase_shift             recording,           format="binary_        "templates",                                   curation
                    whiten                  docker_image=,       folder", folder=)      "spike_amplitudes",                                                exporters
                    correct_motion          ...)                                        "unit_locations",                                                    export_to_phy
                                                                                        "correlograms",                                                      export_report
                                                                                        "template_similarity")                                               export_to_ibl_gui
                                                                                                                                                             to_pynapple_tsgroup

Rules of thumb:

• Preprocessors are lazy — chaining bandpass_filter(...) → common_reference(...) builds a graph; nothing is computed until get_traces() or .save(...) is called.
• Sorters expect a BaseRecording. For heavy sorters call .save(format="binary", n_jobs=..., chunk_duration=...) first so preprocessing is materialized once.
• The SortingAnalyzer is the central hub for everything after sorting. Extensions have a dependency chain (e.g. waveforms needs random_spikes; template_similarity needs templates; spike_amplitudes needs templates; drift quality metrics need spike_locations).
• Quality metrics, template metrics, spike-train metrics are themselves extensions of the SortingAnalyzer.
• Curation returns a new BaseSorting you can re-wrap in a fresh SortingAnalyzer.

5. Quick Start

Minimal end-to-end pipeline, based on spikeinterface/doc/get_started/quickstart.rst:

import spikeinterface.full as si  # heavy but convenient one-shot import

# ---- 0. Parallelism defaults --------------------------------------------
si.set_global_job_kwargs(n_jobs=4, chunk_duration="1s")

# ---- 1. Load a recording (here, a bundled MEArec test file) -------------
local_path = si.download_dataset(remote_path="mearec/mearec_test_10s.h5")
recording, sorting_true = si.read_mearec(local_path)

# ---- 2. Lazy preprocessing chain ----------------------------------------
recording_f   = si.bandpass_filter(recording, freq_min=300, freq_max=6000)
recording_cmr = si.common_reference(recording_f, reference="global", operator="median")

# Materialize once so the sorter reads from disk quickly.
recording_preprocessed = recording_cmr.save(format="binary")

# ---- 3. Run a sorter ----------------------------------------------------
sorting = si.run_sorter(
    sorter_name="tridesclous2",
    recording=recording_preprocessed,
    folder="tdc2_output",
    remove_existing_folder=True,
    # docker_image=True,   # <- uncomment to run an external sorter in Docker
)

# ---- 4. Build a SortingAnalyzer and compute extensions ------------------
analyzer = si.create_sorting_analyzer(
    sorting=sorting,
    recording=recording_preprocessed,
    format="binary_folder",
    folder="analyzer_tdc2",
    sparse=True,
    return_in_uV=True,
)

extensions_to_compute = [
    "random_spikes",
    "waveforms",
    "noise_levels",
    "templates",
    "spike_amplitudes",
    "unit_locations",
    "spike_locations",
    "correlograms",
    ,
]
extension_params = {
    :      {: },
    :     {: },
    :        {: },
    : {: },
}
analyzer.compute(extensions_to_compute, extension_params=extension_params)


qm_params = si.get_default_quality_metrics_params()
qm_params[][]  = 
qm_params[][] = 
analyzer.compute(, metric_params=qm_params)

qm_df = analyzer.get_extension().get_data()


keep_mask = (qm_df[] > ) & (qm_df[] < )
sorting_curated = sorting.select_units(sorting.unit_ids[keep_mask.values])


si.export_report(analyzer, output_folder=)
si.export_to_phy(analyzer, output_folder=)

Reload later with analyzer_reloaded = si.load_sorting_analyzer("analyzer_tdc2").

6. Key Concepts

BaseRecording — the abstract multichannel voltage-trace object returned by every read_* extractor and every preprocessor. Segmented (get_num_segments) with per-segment sample access via get_traces(segment_index, start_frame, end_frame, channel_ids, return_scaled=..., return_in_uV=...). Carries channel ids, sampling frequency, dtype, gain/offset, a probeinterface.Probe, and arbitrary properties/annotations. Serializable via to_dict/from_dict. Sliceable in time (frame_slice) and channels (channel_slice).

BaseSorting — the abstract spike-train container returned by every sorter and every read_*_sorting. Segmented like BaseRecording. Access via get_unit_ids(), get_unit_spike_train(unit_id, segment_index), and vectorized to_spike_vector(). Supports select_units, remove_units, rename_units, plus arbitrary properties (set_property / get_property).

SortingAnalyzer — the post-sorting hub built by create_sorting_analyzer(sorting, recording, format=..., folder=..., sparse=..., return_in_uV=...). Owns a sorting, a recording, a ChannelSparsity, and a dict of computed AnalyzerExtension results. Formats: "memory", "binary_folder", "zarr". Persist with .save_as(...), reload with load_sorting_analyzer(folder).

ChannelSparsity — a per-unit boolean mask over channels (unit × channel). Reduces memory for waveform / template / PCA storage. Built with estimate_sparsity(...) (radius, snr, ptp, best_channels, ...). Extensions honor sparsity automatically. Toggle at analyzer creation with sparse=True/False or pass sparsity=<ChannelSparsity>. A dense analyzer can be sparsified with analyzer.copy(sparsity=...); sparse cannot be trivially densified.

Motion — the drift/motion object living in spikeinterface.core.motion. Estimated by estimate_motion(...) (or correct_motion(...)) and applied to a recording with InterpolateMotionRecording under the hood. Carries displacement, temporal_bins_s, spatial_bins_um, direction. Serializable to disk.

Extension registration — every postprocessing/metric computation is an AnalyzerExtension subclass decorated with register_result_extension(...). This means analyzer.compute("<name>", **params) dispatches to the registered class, records parameters, and persists results in the analyzer's folder under extensions/<name>/. analyzer.get_extension("<name>").get_data() returns the computed data. Extensions know their dependencies; SpikeInterface raises if you request one whose parents are missing.

Lazy vs. computed — preprocessors are lazy: they only wrap the graph and compute on demand. Extensions on a SortingAnalyzer are computed (and persisted) at .compute(...) time. To force a preprocessing chain to disk, call recording.save(format="binary", folder=..., n_jobs=..., chunk_duration=...) — this reads once and lets downstream operations reuse the cached traces.

7. Common Pitfalls

1. return_scaled is deprecated in favor of return_in_uV. Use return_in_uV=True on create_sorting_analyzer(...) and on recording.get_traces(...). Setting return_scaled=True still works but prints a deprecation warning; setting both is an error.
2. Sparse vs. dense. create_sorting_analyzer(..., sparse=True) is the default and required by many extensions to stay memory-safe. Passing dense waveforms to a Neuropixels-scale recording will exhaust RAM. If you need dense outputs (e.g. for plot_unit_templates across all channels), either build a dense analyzer or estimate sparsity with a large radius.
3. n_jobs and chunk_duration are cross-cutting. They are read from set_global_job_kwargs(...) unless overridden per-call. Excess n_jobs combined with mp_context="fork" on Linux (default) can OOM. For Windows or notebook use, prefer mp_context="spawn". For sorters that already parallelize internally (kilosort4, spykingcircus2) keep n_jobs moderate.
4. Extension dependency chain. Requesting spike_amplitudes without templates, template_similarity without templates, waveforms without random_spikes, or drift-based quality metrics without spike_locations will raise. Either compute in the right order or hand analyzer.compute([...]) a list and let it resolve.
5. Sorter installation. si.installed_sorters() shows what actually resolves in your env; si.available_sorters() shows everything SpikeInterface knows about. Kilosort2/2.5/3/4 need a working GPU + MATLAB (2/2.5/3) or PyTorch (4). If a sorter is not natively installed, use docker_image=True or singularity_image=True — SpikeInterface pulls a prebuilt image from spikeinterface/<sorter>-compiled-base and runs it transparently.
6. Docker/Singularity fallbacks require the host tool. Docker requires the Docker daemon + docker Python bindings; Singularity requires the singularity/apptainer binary + spython. On HPC prefer Singularity. Container mode auto-installs the current spikeinterface version inside the container ( when running from a source checkout).

8. Installation and Import Shortcuts

Install with extras for a full-featured environment (quotes required in zsh):

pip install "spikeinterface[full]"

# Add interactive widget backends (ipywidgets, figpack, ephyviewer, spikeinterface_gui, ...).
# `sortingview` is still installable but is a legacy alias for figpack — prefer figpack for new work.
pip install "spikeinterface[full,widgets]"

# Development install from source
git clone https://github.com/SpikeInterface/spikeinterface.git
cd spikeinterface
pip install -e .

Two supported import styles:

# Style A: one flat namespace (heavy import, ideal for notebooks)
import spikeinterface.full as si
recording = si.read_spikeglx("/data/npx_run/")
recording = si.bandpass_filter(recording, freq_min=300, freq_max=6000)
sorting   = si.run_sorter("kilosort4", recording, docker_image=True)
analyzer  = si.create_sorting_analyzer(sorting, recording, format="binary_folder", folder="out")

# Style B: explicit per-submodule imports (lighter, preferred in production)
import spikeinterface           as si
import spikeinterface.extractors     as se
import spikeinterface.preprocessing  as spre
import spikeinterface.sorters        as ss
import spikeinterface.postprocessing as spost
import spikeinterface.metrics.quality as sqm
import spikeinterface.curation       as scur
import spikeinterface.comparison     as sc
import spikeinterface.widgets        as sw
import spikeinterface.exporters      as sexp
import spikeinterface.generation     as sgen
import spikeinterface.benchmark      as sbench

spikeinterface.full re-exports (in this order) core, extractors, sorters, preprocessing, postprocessing, metrics, curation, comparison, widgets, exporters, generation, benchmark — see src/spikeinterface/full.py.