data-source-priority

name	data-source-priority
description	Ensure Alpaca API is used for quality data, not yfinance fallback. Trigger when: (1) crypto volume filter fails unexpectedly, (2) zero-volume bars in data, (3) API key configuration issues.
author	Claude Code
date	2024-12-28T00:00:00.000Z

Data Source Priority - Alpaca vs yfinance (v2.5.0)

Experiment Overview

Item	Details
Date	2024-12-28
Goal	Fix crypto selection failing due to poor data quality
Environment	alpaca_trading/data/fetcher.py, training notebook
Status	Success - v2.5.0 now fails early

Context

User reported ALL 18 crypto symbols failing selection filters:

BTCUSD: failed price (max_price $10k but BTC is $87k)
ETHUSD: failed data_quality (51% zero-volume bars)
Others: failed volume, trading_status

Root Cause: Alpaca API keys weren't configured, so DataFetcher fell back to yfinance. yfinance crypto data has:

• ~50% zero-volume bars (data gaps)
• Missing volume data on many bars
• Causes ALL filters to fail

v2.5.0 Solution: Fail Fast

OLD BEHAVIOR (v2.4.x): Warned about yfinance but continued anyway, wasting time debugging filter failures.

NEW BEHAVIOR (v2.5.0): Training notebook FAILS IMMEDIATELY if crypto is enabled without Alpaca API:

# In training notebook cell-16
alpaca_enabled = data_fetcher._use_alpaca_data

if not alpaca_enabled and selection_config.crypto.enabled:
    raise ValueError(
        'CRYPTO TRAINING REQUIRES ALPACA API. '
        'yfinance crypto data has ~50% missing volume. '
        'Set API keys or disable crypto training.'
    )

Clear Error Message

When crypto is enabled without Alpaca API:

======================================================================
ERROR: CRYPTO TRAINING REQUIRES ALPACA API
======================================================================

yfinance crypto data is unusable for training:
  - ~50% zero-volume bars
  - Causes all symbols to fail volume/data_quality filters
  - Training on bad data produces bad models

FIX OPTIONS:
  1. Set Alpaca API keys in Colab Secrets:
     - APCA_API_KEY_ID = your_key
     - APCA_API_SECRET_KEY = your_secret

  2. Set API_KEYS_FILE in previous cell:
     API_KEYS_FILE = "/content/Alpaca_trading/API_key_500Paper.txt"

  3. Disable crypto and train only equities:
     selection_config.crypto.enabled = False
     selection_config.equities.enabled = True

Failed Attempts (Critical)

Attempt	Why it Failed	Lesson Learned
Lower volume consistency to 30%	Masks real problem, still fails	Fix data source, not thresholds
Skip data quality filter for crypto	Still fails price/trading_status	Garbage in = garbage out
Simplify all crypto filters	Works but produces bad models	Use quality data, not workarounds
Just warn about yfinance	User ignores warning, wastes time	FAIL FAST is better

Data Quality Comparison

Data Source	Volume Data	Zero-Volume Bars	Crypto Support
Alpaca API	Complete	<1%	Excellent
yfinance	50% missing	~50%	Unusable

API Key Configuration

For Google Colab (RECOMMENDED)

1. Go to Colab Secrets (key icon in left sidebar)
2. Add APCA_API_KEY_ID = your API key
3. Add APCA_API_SECRET_KEY = your secret key
4. Enable access to notebook

For Training Notebook

# Cell 14: Option 1 - Environment variables (recommended)
# Keys are read from Colab Secrets automatically

# Cell 15: Option 2 - Keys file (after unzipping repo)
API_KEYS_FILE = '/content/Alpaca_trading/API_key_500Paper.txt'

For Local Development

# Add to ~/.bashrc or ~/.zshrc
export APCA_API_KEY_ID="your_key"
export APCA_API_SECRET_KEY="your_secret"

Diagnostic Check

from alpaca_trading.data.fetcher import DataFetcher
fetcher = DataFetcher(keys_file=API_KEYS_FILE)

# This is checked BEFORE selection in v2.5.0
if not fetcher._use_alpaca_data:
    print('Alpaca API: NOT AVAILABLE')
    # Notebook will fail here if crypto enabled
else:
    print('Alpaca API: ENABLED')

Key Insights

1. Don't work around bad data - Fix the data source
2. Fail fast, fail loud - Silent fallbacks waste debugging time
3. yfinance is equities-only - Acceptable for stocks, unusable for crypto
4. Environment variables are best - Work everywhere, no path issues
5. Check logs for "yfinance fetched" - This means you're using bad data

Files Modified (v2.5.0)

notebooks/training.ipynb:
  - Cell 16: Added fail-fast check before symbol selection
  - Cell 0: Updated header to v2.5.0 with changelog

alpaca_trading/selection/filters/hard_filters.py:
  - apply_hard_filters(): Simplified crypto path (skip yfinance-specific checks)

References

• alpaca_trading/data/fetcher.py: DataFetcher implementation
• notebooks/training.ipynb: Training notebook with fail-fast check
• Alpaca API docs: https://docs.alpaca.markets/docs/
• Skill: reward-function-hold-bias - Related v2.5.0 fix