scikit-video

# scikit-video - Scientific Video Processing scikit-video simplifies the complex world of video codecs and containers by providing a consistent NumPy-based interface. It allows for the calculation of motion vectors, video quality assessment (VQA), and seamless integration with the rest of the scientific Python stack. ## When to Use - Reading and writing video files in various formats (MP4, AVI, MKV) via FFmpeg. - Extracting specific frames or segments from long videos without loading them entirely into memory. - Calculating motion estimation (Block Matching, Optical Flow). - Measuring video quality (PSNR, SSIM, VIF, NIQE). - Generating video datasets for machine learning. - Visualizing temporal changes in pixel data (e.g., scientific recordings). - Handling raw YUV data streams. ## Reference Documentation **Official docs**: http://www.scikit-video.org/ **GitHub**: https://github.com/scikit-video/scikit-video **Search patterns**: `skvideo.io.vread`, `skvideo.io.FFmpegReader`, `skvideo.motion`, `skvideo.measure` ## Core Principles ### Video as 4D Arrays A video is represented as a NumPy array with shape (T, H, W, C): - **T**: Time (number of frames) - **H**: Height - **W**: Width - **C**: Channels (usually 3 for RGB) ### FFmpeg Backend Scikit-video does not contain its own codecs; it is a bridge to FFmpeg. You must have FFmpeg installed on your system for `skvideo.io` to function. ### Generators for Large Data For long videos, scikit-video provides generator-based readers (`vreader`) to process frames one by one, preventing RAM exhaustion. ## Quick Reference ### Installation ```bash pip install scikit-video # Note: Ensure ffmpeg is in your system PATH ``` ### Standard Imports ```python import skvideo.io import skvideo.motion import skvideo.measure import numpy as np ``` ### Basic Pattern - Read and Inspect ```python import skvideo.io # 1. Read the whole video into a NumPy array # Shape: (frames, height, width, 3) video_data = skvideo.io.vread("experiment.mp4") # 2. Get basic info n_frames, height, width, channels = video_data.shape print(f"FPS: {n_frames / 10}, Resolution: {width}x{height}") # 3. Access a specific frame frame_10 = video_data[10] ``` ## Critical Rules ### ✅ DO - **Use vreader for large files** - Always use generator-based reading for high-resolution or long videos. - **Set num_frames** - If you know the number of frames you need, specify it to avoid unnecessary scanning. - **Check FFmpeg path** - Use `skvideo.setFFmpegPath()` if FFmpeg is installed in a non-standard location. - **Normalize for Metrics** - Ensure pixel values are in the range expected by `skvideo.measure` (usually [0, 255] for uint8). - **Use vwrite for simple output** - It handles the complex FFmpeg command-line arguments for you. - **Consider YUV** - When working with raw transmission data, use the specific YUV reading capabilities. ### ❌ DON'T - **Load 4K video with vread** - A 1-minute 4K video will exceed most RAM capacities. - **Ignore the inputdict and outputdict** - These allow you to pass specific flags to FFmpeg (like bitrate, pixel format, or codec). - **Assume RGB order** - Always verify the channel order after reading, especially if using external codecs. - **Process video without Denoising** - Video noise can ruin motion estimation; apply spatial or temporal filters first. ## Anti-Patterns (NEVER) ```python import skvideo.io # ❌ BAD: Loading a massive file at once # video = skvideo.io.vread("huge_4k_recording.mp4") # CRASH! # ✅ GOOD: Processing frame by frame reader = skvideo.io.vreader("huge_4k_recording.mp4") for frame in reader: # Process frame pass # ❌ BAD: Manual frame writing in a loop with manual codec setup # (Fragile and complex) # ✅ GOOD: Use FFmpegWriter writer = skvideo.io.FFmpegWriter("output.mp4") for frame in processed_frames: writer.writeFrame(frame) writer.close() # ❌ BAD: Relying on system default FFmpeg without checking # ✅ GOOD: Verify backend # print(skvideo._HAS_FFMPEG) ``` ## Reading and Writing (skvideo.io) ### Advanced Video I/O ```python import skvideo.io # 1. Reading with specific FFmpeg options input_parameters = { "-ss": "00:00:10", # Start at 10 seconds "-t": "5" # Duration 5 seconds } video = skvideo.io.vread("video.mp4", inputdict=input_parameters) # 2. Writing with specific bitrate and codec output_parameters = { "-vcodec": "libx264", "-b:v": "5000k", # 5 Mbps bitrate "-pix_fmt": "yuv420p" } skvideo.io.vwrite("output.mp4", video, outputdict=output_parameters) ``` ## Motion Estimation (skvideo.motion) ### Calculating Movement ```python from skvideo.motion import blockMotion from skvideo.io import vread # Load two consecutive frames video = vread("video.mp4") frame1 = video[0] frame2 = video[1] # Block matching algorithm # Returns motion vectors for each block motion_vectors = blockMotion(frame1, frame2, method='DS', mbSize=16) # motion_vectors shape: (H/mbSize, W/mbSize, 2) # The last dimension contains (dy, dx) offsets ``` ## Video Quality Assessment (skvideo.measure) ### Measuring Degradation ```python from skvideo.measure import psnr, ssim, mse # Compare original and compressed video original = vread("original.mp4") distorted = vread("compressed.mp4") # Calculate metrics frame by frame psnr_scores = psnr(original, distorted) ssim_scores = ssim(original, distorted) print(f"Average PSNR: {np.mean(psnr_scores)}") ``` ## Datasets and Utilities ### Using Internal Datasets ```python import skvideo.datasets # Load a built-in sample video (useful for testing) path = skvideo.datasets.bigbuckbunny() reader = skvideo.io.vreader(path) ``` ## Practical Workflows ### 1. Simple Background Subtraction Pipeline ```python def extract_background(video_path): """Calculates the static background of a video using the median.""" reader = skvideo.io.vreader(video_path) frames = [] # Sample every 10th frame to save memory for i, frame in enumerate(reader): if i % 10 == 0: frames.append(frame) if len(frames) > 50: break # Background is the median of frames background = np.median(np.array(frames), axis=0).astype(np.uint8) return background # Usage # bg = extract_background("security_cam.mp4") ``` ### 2. Video Stabilization (Frame Alignment) ```python def stabilize_frames(video_array): """Very basic stabilization using motion vectors.""" stabilized = [video_array[0]] for i in range(1, len(video_array)): motion = skvideo.motion.blockMotion(video_array[i-1], video_array[i]) avg_motion = np.mean(motion, axis=(0, 1)) # Global drift # Translate frame back (Simplified logic) # Use scipy.ndimage.shift for actual translation ... ``` ### 3. Automated Video Quality Report ```python def generate_vqa_report(ref_path, test_path): ref = skvideo.io.vread(ref_path) test = skvideo.io.vread(test_path) report = { "MSE": np.mean(skvideo.measure.mse(ref, test)), "PSNR": np.mean(skvideo.measure.psnr(ref, test)), "SSIM": np.mean(skvideo.measure.ssim(ref, test)) } return report ``` ## Performance Optimization ### Using vreader with Multi-threading If you are doing heavy processing on each frame, use a queue-based multi-threading approach to keep the FFmpeg pipe full. ### Efficient Slicing Instead of `vread` and then slicing, use FFmpeg's seek and duration flags via `inputdict` to only read the data you need from the disk. ## Common Pitfalls and Solutions ### FFmpeg Not Found Scikit-video relies on the ffmpeg executable. ```python # ✅ Solution: Manually set the path if it's not in environment import skvideo skvideo.setFFmpegPath("C:/ffmpeg/bin") ``` ### Color Space Mismatches Some videos are stored in YUV422 or YUV444. Scikit-video converts these to RGB by default. ```python # ❌ Problem: Colors look washed out or incorrect # ✅ Solution: Specify the pixel format in inputdict reader = skvideo.io.vreader("video.mp4", inputdict={"-pix_fmt": "yuv420p"}) ``` ### Out of Memory (OOM) Errors Even with `vreader`, if you store all frames in a list, you will run out of memory. ```python # ❌ Problem: frames.append(frame) in a loop # ✅ Solution: Process and save to disk, or clear the list periodically. ``` scikit-video brings the power of FFmpeg into the NumPy world. By abstracting the complexities of video containers and providing scientific analysis tools like motion estimation and quality metrics, it is an essential tool for any researcher working with temporal image data.