limelight3a

name	limelight3a
description	Expert guidance for using Limelight 3A vision camera in FTC robotics. Use when working with AprilTag detection, robot localization, MegaTag, pipeline configuration, coordinate transformations, or vision-based targeting.

Limelight 3A FTC Vision Skill

This skill provides comprehensive guidance for using the Limelight 3A vision camera in FTC robotics for AprilTag detection, robot localization, and vision-based targeting.

Overview

The Limelight 3A is a compact USB vision camera optimized for FTC with:

• OV5647 sensor: 640×480 @ 90 FPS
• Field of View: 54.5° horizontal, 42° vertical
• AprilTag tracking with MegaTag/MegaTag2 robot localization
• 10 hot-swappable pipelines for different vision tasks
• Web-based configuration UI at http://limelight.local:5801
• USB-C connection to Control Hub USB 3.0 port
• No LED illumination (rely on ambient light or external LEDs)

Power: 4.1V-5.75V via USB, max 4W consumption Weight: 0.20 lb Dimensions: 2.839" × 1.894" × 0.661"

FTC API Overview

Initialization

import com.qualcomm.hardware.limelightvision.Limelight3A
import com.qualcomm.hardware.limelightvision.LLResult
import com.qualcomm.hardware.limelightvision.LLResultTypes

// In init()
val limelight = hardwareMap.get(Limelight3A::class.java, "limelight")
limelight.pipelineSwitch(0)     // Switch to pipeline 0
limelight.setPollRateHz(100)    // Poll 100 times/sec
limelight.start()               // Start processing

Key Limelight3A Methods

Method	Description
pipelineSwitch(int)	Switch between pipelines 0-9
getLatestResult()	Get current LLResult with all detection data
updateRobotOrientation(double yaw)	Provide IMU heading for MegaTag2 fusion
updatePythonInputs(double[])	Send data to Python SnapScript pipelines
setPollRateHz(int)	Set polling frequency (default 100 Hz)
start()	Begin vision processing
captureSnapshot(String)	Save debug images
deleteSnapshots()	Clear stored snapshots

LLResult Object

Always check validity first:

val result = limelight.latestResult
if (result != null && result.isValid) {
    // Use result data
}

IMPORTANT: getLatestResult() always returns an LLResult object, but isValid() will be false if no valid targets are detected. Always check isValid() before accessing data.

Basic Targeting Data:

result.tx              // Horizontal angle to target (degrees)
result.ty              // Vertical angle to target (degrees)
result.ta              // Target area (0-100% of image)
result.isValid         // true if data is valid
result.getStaleness()  // Age of data in milliseconds
result.pipelineIndex   // Current pipeline number

Robot Localization (MegaTag):

result.botpose              // Pose3D: MegaTag 1 position
result.botpose_MT2          // Pose3D: MegaTag 2 with IMU fusion
result.botposeTagCount      // Number of AprilTags used for pose

// Access pose components:
val pose = result.botpose
pose.position.x             // X position (meters)
pose.position.y             // Y position (meters)
pose.position.z             // Z position (meters)
pose.orientation.yaw        // Yaw angle (degrees)
pose.orientation.pitch      // Pitch angle (degrees)
pose.orientation.roll       // Roll angle (degrees)

AprilTag Detection:

result.fiducialResults      // List<FiducialResult>

for (tag in result.fiducialResults) {
    tag.fiducialId          // AprilTag ID
    tag.targetXDegrees      // Horizontal angle to tag
    tag.targetYDegrees      // Vertical angle to tag
    tag.targetArea          // Tag area percentage
    tag.robotPose           // Pose3D of robot relative to tag
    tag.targetPose          // Pose3D of tag relative to camera
}

Other Result Types:

result.colorResults         // Color blob detection
result.barcodeResults       // QR code/barcode data
result.classifierResults    // Neural network classifications
result.detectorResults      // Object detection bounding boxes
result.pythonOutput         // Data from Python pipelines (double[])

Pipeline Configuration

Accessing Web UI

1. Connect Limelight via USB to Control Hub
2. Connect Driver Station to Control Hub WiFi
3. Navigate to http://limelight.local:5801 in browser
4. Or use Limelight Hardware Manager application

Pipeline Types

Limelight 3A supports 10 unique pipelines (0-9):

1. AprilTag Pipelines - Fiducial marker detection
2. Color Pipelines - Color blob tracking (90 FPS)
3. Neural Network Pipelines - CPU inference (~10 FPS)
4. Python Pipelines - Custom OpenCV processing
5. Barcode Pipelines - QR code/barcode reading

AprilTag Pipeline Configuration

Input Tab Settings:

Setting	Recommended Value	Notes
Resolution	640×480	Higher = better 3D accuracy, lower FPS
Exposure	Low (start 1-2 ms)	Minimize motion blur
Sensor Gain	15	Increase brightness without longer exposure
Black Level	0	Reduce arena lights/bright spots

Fiducial Tab Settings:

Setting	Value	Notes
Family	AprilTag Classic 36h11	FTC standard
Tag Size	Varies by game	DECODE: 50mm for samples, etc.
Detector Downscale	1.0-2.0	Higher = more FPS, less range
Quality Threshold	Default	Filters spurious detections
ID Filters	Comma-separated IDs	e.g., "1,2,3,21,22,23"

3D Settings:

• Enable "Full 3D" mode for robot localization
• Configure camera pose relative to robot center
• Upload field map with AprilTag locations

Pipeline Switching

Switch pipelines dynamically for different tasks:

// Pipeline 0: AprilTag localization
limelight.pipelineSwitch(0)

// Pipeline 1: Sample detection (color)
limelight.pipelineSwitch(1)

// Pipeline 2: Specimen detection
limelight.pipelineSwitch(2)

Robot Localization with MegaTag

MegaTag vs MegaTag2

MegaTag 1 (botpose):

• Uses only AprilTag detections
• No IMU fusion
• Works with 1+ tags visible
• More stable with multiple tags

MegaTag2 (botpose_MT2):

• Fuses AprilTag data with IMU
• Requires updateRobotOrientation(yaw) calls
• More accurate with IMU fusion
• Resilient to temporary tag loss

Recommendation: Use MegaTag2 for best results in FTC.

Setting Up Localization

1. Configure Camera Pose in Web UI:

• Navigate to "3D" tab in AprilTag pipeline
• Enter camera position relative to robot center
• Example: X=0, Y=0, Z=0.2m (20cm above center)
• Enter camera orientation (pitch, yaw, roll)

2. Upload Field Map:

• Use pre-made FTC field maps from Limelight
• Or create custom map with AprilTag positions
• Upload via "Field Map" button in web UI

3. Enable in Code:

// Update IMU heading every loop
limelight.updateRobotOrientation(imu.robotYawPitchRollAngles.getYaw(AngleUnit.DEGREES))

val result = limelight.latestResult
if (result != null && result.isValid) {
    if (result.botposeTagCount >= 2) {  // Use 2+ tags for reliability
        val pose = result.botpose_MT2    // MegaTag2 with IMU fusion

        // Use pose for odometry correction
        val x = pose.position.x
        val y = pose.position.y
        val heading = Math.toRadians(pose.orientation.yaw)
    }
}

Coordinate System Transformations

Limelight Coordinate System (from camera perspective):

• X-axis: Right is positive
• Y-axis: Down is positive
• Z-axis: Forward (away from camera) is positive

FTC Field Coordinate System (standard):

• X-axis: Varies by alliance
• Y-axis: Varies by alliance
• Heading: Varies by starting orientation

Common Transformation Pattern (from this codebase):

// Limelight returns pose in field coordinates
val botX = -botpose.position.y  // Swap and negate Y
val botY = botpose.position.x   // Use X
val botRot = ((botpose.orientation.yaw * Math.PI) / 180.0) - Math.PI  // Convert to radians, adjust

Important: Always test coordinate transformations empirically. Place robot at known position, read Limelight pose, and adjust transformation until it matches.

Integration Patterns

Pattern 1: Motif Detection (Game Element Patterns)

Detect specific AprilTag IDs to determine game element configuration:

enum class Motif {
    GPP,  // Green-Purple-Purple
    PGP,  // Purple-Green-Purple
    PPG   // Purple-Purple-Green
}

fun detectMotif(fiducialResults: List<LLResultTypes.FiducialResult>): Motif {
    var currentMotif = Motif.GPP  // Default

    for (tag in fiducialResults) {
        currentMotif = when (tag.fiducialId) {
            21 -> Motif.GPP
            22 -> Motif.PGP
            23 -> Motif.PPG
            else -> currentMotif
        }
    }

    return currentMotif
}

Pattern 2: Continuous Localization Update

Update robot odometry with vision data:

class LimelightIO(val limelight: Limelight3A) {
    var localization = Pose2D(DistanceUnit.METER, 0.0, 0.0, AngleUnit.RADIANS, 0.0)
        private set

    var localizationTags = 0
        private set

    fun run(imuYaw: Double) {
        // Provide IMU for MegaTag2
        limelight.updateRobotOrientation(imuYaw)

        val result = limelight.latestResult
        if (result != null && result.isValid) {
            localizationTags = result.botposeTagCount

            // Only use pose if 2+ tags visible (more reliable)
            if (result.botposeTagCount >= 2) {
                updateBotPosition(result.botpose_MT2)
            }
        }
    }

    private fun updateBotPosition(botpose: Pose3D) {
        // Transform from Limelight to field coordinates
        val botX = -botpose.position.y
        val botY = botpose.position.x
        val botRot = ((botpose.orientation.yaw * Math.PI) / 180.0) - Math.PI

        localization = Pose2D(DistanceUnit.METER, botX, botY, AngleUnit.RADIANS, botRot)
    }
}

Pattern 3: Vision-Assisted Targeting

Use tx/ty for alignment assistance:

fun isAlignedWithTarget(result: LLResult, toleranceDegrees: Double = 2.0): Boolean {
    if (!result.isValid) return false
    return abs(result.tx) < toleranceDegrees
}

fun getSteeringCorrection(result: LLResult, kP: Double = 0.05): Double {
    if (!result.isValid) return 0.0
    return -result.tx * kP  // Proportional control
}

Pattern 4: Odometry Fusion with Vision

Combine vision with odometry using weighted averaging or Kalman filter:

fun fuseVisionWithOdometry(
    odomPose: Pose2D,
    visionPose: Pose2D,
    visionWeight: Double = 0.15  // 15% vision, 85% odometry
): Pose2D {
    val fusedX = odomPose.x * (1 - visionWeight) + visionPose.x * visionWeight
    val fusedY = odomPose.y * (1 - visionWeight) + visionPose.y * visionWeight

    // Heading fusion (handle wrapping)
    val odomHeading = odomPose.heading
    val visionHeading = visionPose.heading
    val headingDiff = normalizeAngle(visionHeading - odomHeading)
    val fusedHeading = normalizeAngle(odomHeading + headingDiff * visionWeight)

    return Pose2D(DistanceUnit.METER, fusedX, fusedY, AngleUnit.RADIANS, fusedHeading)
}

Best Practices

Performance Optimization

1. Use appropriate resolution: 320×240 for 2D detection (90 FPS), 640×480 for 3D localization
2. Minimize exposure: Reduce motion blur during robot movement (start at 1-2 ms)
3. Increase detector downscale: Trades range for framerate (1.5-2.0 works well)
4. Filter by tag count: Only use pose estimates with 2+ tags for reliability
5. Set poll rate: 100 Hz is good balance for FTC (don't exceed loop rate)

Reliability

1. Always check isValid() before accessing result data
2. Check staleness: Reject old data (getStaleness() > 50ms)
3. Validate tag count: More tags = more reliable pose (botposeTagCount >= 2)
4. Handle missing targets: Have fallback behavior when no tags detected
5. Test transformations: Empirically verify coordinate transformations

Camera Mounting

1. Height: Mount above or below tag height for better detection
2. Angle: Slight tilt (10-20°) prevents head-on ambiguity
3. Stable mount: Minimize vibration and movement
4. USB routing: Ensure USB-C cable won't disconnect during movement
5. Field of view: Position to see multiple tags simultaneously

Debugging

1. Use web UI: Live view shows detections in real-time
2. Capture snapshots: Save problem frames for offline analysis
3. Check telemetry: Monitor tx, ty, tag count, staleness
4. Verify field map: Ensure AprilTag positions match physical field
5. Test camera pose: Verify robot-to-camera transform is correct

Common Issues and Solutions

Issue: No AprilTags Detected

Solutions:

• Check lighting (too bright or too dark)
• Reduce exposure to minimize motion blur
• Increase sensor gain for low light
• Verify tag size configuration matches physical tags
• Check ID filters aren't excluding your tags
• Ensure tags are flat and not damaged

Issue: Unreliable Pose Estimates

Solutions:

• Only use poses with 2+ tags (botposeTagCount >= 2)
• Check camera pose configuration in web UI
• Verify field map matches physical tag locations
• Use MegaTag2 with IMU fusion (updateRobotOrientation())
• Reject poses with low tag count or high staleness
• Implement outlier rejection (check if pose jumped >1m)

Issue: Wrong Coordinate Transformation

Solutions:

• Test transformation empirically (place robot at known position)
• Check coordinate system documentation
• Verify camera orientation in robot pose config
• Use telemetry to debug X, Y, heading separately
• Consider alliance-specific transformations

Issue: Low Frame Rate

Solutions:

• Use lower resolution (320×240 instead of 640×480)
• Increase detector downscale (1.5-2.0)
• Reduce number of tags in ID filter
• Use simpler pipeline type (AprilTag is faster than neural nets)
• Check USB connection quality

Issue: Tags Detected But Wrong IDs

Solutions:

• Increase quality threshold to filter spurious detections
• Use ID filters to restrict to expected tags
• Check tag family setting (should be 36h11 for FTC)
• Ensure tags are correctly printed/sized
• Improve lighting conditions

Integration with This Codebase

Current Architecture

LimelightIO.kt - Abstraction layer:

class LimelightIO(limelight3A: Limelight3A, physicalHardware: HardwareIO) {
    // Manages:
    // - Motif detection (GPP, PGP, PPG)
    // - Robot localization from AprilTags
    // - Coordinate transformations
    // - IMU fusion via updateRobotOrientation()
}

Limelight.kt - Subsystem:

class Limelight(robot: Robot): SubsystemBase() {
    // Periodic updates:
    // - Displays detected motif in telemetry
    // - Can provide vision measurements to odometry
}

HardwareIO.kt - Hardware interface:

override fun limelightDetectedMotif(): Limelight.Motif
override fun limelightLocalizationAprilTagCount(): Int
override fun limelightLocalization(): Pose2D

Usage in Autonomous

// In FarAuto.kt
val motif = robot.hardware.limelightDetectedMotif()
ShootMotif(robot.spiceRack, motif)

Usage in TeleOp

Display targeting info:

telemetry.addData("Limelight Tags", hardware.limelightLocalizationAprilTagCount())
telemetry.addData("Motif", hardware.limelightDetectedMotif())

Resources

• Official Docs: https://docs.limelightvision.io/docs/docs-limelight/getting-started/limelight-3a
• FTC API Guide: https://docs.limelightvision.io/docs/docs-limelight/apis/ftc-programming
• AprilTag Detection: https://docs.limelightvision.io/docs/docs-limelight/pipeline-apriltag/apriltags
• Robot Localization: https://docs.limelightvision.io/docs/docs-limelight/pipeline-apriltag/apriltag-robot-localization
• Pipeline Setup: https://docs.limelightvision.io/docs/docs-limelight/getting-started/pipelines
• Web UI: http://limelight.local:5801

Quick Reference

Task	Code
Initialize	hardwareMap.get(Limelight3A::class.java, "limelight")
Switch pipeline	limelight.pipelineSwitch(0)
Get result	limelight.latestResult
Check validity	result.isValid
Target angle	result.tx, result.ty
Robot pose	result.botpose_MT2
Tag count	result.botposeTagCount
Update IMU	updateRobotOrientation(yaw)
AprilTag IDs	result.fiducialResults
Start processing	limelight.start()

Remember:

• Always check result.isValid before accessing data
• Use 2+ tags for reliable localization
• Update IMU heading for MegaTag2
• Test coordinate transformations empirically
• Access web UI at http://limelight.local:5801