Use when adding a custom or external data source to a QuantConnect/LEAN algorithm. Triggers: custom data reader, external dataset, py`PythonData`cs`BaseData`, CSV, JSON, XML, ZIP, REST endpoint, Object Store, linked data, unlinked signals, custom universes, or local files for a QC strategy. Skip existing QC datasets subscribed with py`add_data`cs`AddData`, unless writing a custom reader.
Instalar com Codex ou Claude Copie este prompt, cole no Codex, Claude ou outro assistente e deixe que ele revise a página da skill e instale para você.
Um comando direto ignora o prompt de revisão. Verifique a origem antes de executá-lo.
Use when adding a custom or external data source to a QuantConnect/LEAN algorithm. Triggers: custom data reader, external dataset, py`PythonData`cs`BaseData`, CSV, JSON, XML, ZIP, REST endpoint, Object Store, linked data, unlinked signals, custom universes, or local files for a QC strategy. Skip existing QC datasets subscribed with py`add_data`cs`AddData`, unless writing a custom reader.
Custom Data in QuantConnect / LEAN
Build a custom reader, wire it into pymain.pycsMain.cs, and verify rows load. Keep the reader in pysnake_case.pycsPascalCase.cs. Use pyadd_datacsAddData only after the reader type is defined or imported.
1. Identify the data shape
Source: remote file URL, REST endpoint for live mode, local file to upload, or existing Object Store key.
Format: CSV, JSON, XML, ZIP, or line-based text. For JSON, decide whether the payload is one record, newline-delimited records, or an array that unfolds into many records.
Scope: unlinked standalone signal, linked stream for an existing QC asset, one symbol per subscription, or custom universe with many symbols per date.
Coverage: first date, last date, resolution, time zone, and whether live/backtest sources differ.
Fields: timestamp, numeric fields, pyvaluecsValue, plus string, bool, category, URL, and nullable fields that need typed properties or dynamic fields.
Storage: for Cloud Object Store use lean cloud object-store set <key> <path> from an initialized Lean workspace; for local backtests use lean object-store set or copy into the workspace storage folder with the intended key path.
2. Choose the reader pattern
Regular unlinked: standalone symbol such as weather or macro data; remote file or Object Store.
Regular linked: data describes an existing security; subscribe to the security first, then pass its Symbol to pyadd_datacsAddData.
Dual source: branch on pyis_live_modecsisLiveMode only when backtests use files and live trading polls REST.
Unfolding collection: JSON array or one line yields many records; use pyFileFormat.UNFOLDING_COLLECTIONcsFileFormat.UnfoldingCollection.
ZIP: when a .zip file contains CSV rows, subscribe to the .zip key directly with pyFileFormat.CSVcsFileFormat.Csv; LEAN unpacks the stream before calling pyreadercsReader, so the parser is the same as the regular CSV example. pyFileFormat.ZIP_ENTRY_NAMEcsFileFormat.ZipEntryName is for zip entry names, not parsing row content.
Universe: dated file emits symbols; verify selection counts instead of single-symbol history.
Do not use pytrycstry / pyexceptcscatch to hide parser errors. Return pyNonecsnull only for known skipped records: blanks, headers, comments, or malformed optional rows the user explicitly wants ignored.
3. Minimal reader
Replace class name, key/URL, date parsing, fields, and value index. For remote files, switch to pySubscriptionTransportMedium.REMOTE_FILEcsSubscriptionTransportMedium.RemoteFile.
using QuantConnect.Data;
using System;
using System.Globalization;
publicclassMyCustomData : BaseData
{
publicdecimal Signal { get; set; }
publicoverride SubscriptionDataSource GetSource(SubscriptionDataConfig config, DateTime date, bool isLiveMode)
{
returnnew SubscriptionDataSource("custom-data/my-dataset.csv", SubscriptionTransportMedium.ObjectStore);
}
publicoverride BaseData Reader(SubscriptionDataConfig config, string line, DateTime date, bool isLiveMode)
{
if (string.IsNullOrWhiteSpace(line) || !char.IsDigit(line[0]))
returnnull;
var csv = line.Split(',');
var data = new MyCustomData { Symbol = config.Symbol };
data.Time = DateTime.ParseExact(csv[0], "yyyy-MM-dd", CultureInfo.InvariantCulture);
data.EndTime = data.Time.AddDays(1);
data.Signal = decimal.Parse(csv[1], CultureInfo.InvariantCulture);
data.Value = data.Signal;
return data;
}
}
4. Subscribe, trade, verify
Store custom symbols and read from the custom symbol, not the underlying asset. For non-universe readers, add one history row-count check in pyon_end_of_algorithmcsOnEndOfAlgorithm.
For linked custom data, subscribe to the asset first, pass its Symbol to pyadd_datacsAddData, trade the asset symbol, and read the custom symbol only for signals.
self._asset = self.add_equity("AAPL", Resolution.DAILY).symbol
self._signal = self.add_data(MyCustomData, self._asset, Resolution.DAILY).symbol
ifself._signal in data and data[self._signal].value > 0:
self.set_holdings(self._asset, 1)
JSON: use pyimport jsoncsusing Newtonsoft.Json.Linq;, parse named fields, preserve non-numeric fields, set pyvaluecsValue from the requested numeric signal, and fail loudly on unexpected shape.
ZIP: when the .zip is already in Object Store, do not re-upload it, extract it to another Object Store key, or change the original key. Point SubscriptionDataSource at the .zip key and set pyFileFormat.CSVcsFileFormat.Csv if the contents are CSV; keep the same pyreadercsReader implementation as a normal CSV file. Use pyFileFormat.ZIP_ENTRY_NAMEcsFileFormat.ZipEntryName only when the data points are zip entry names.
Research notebooks are different: pyqb.object_store.read_bytes returns the raw zip bytes, so unzip them before loading the inner file.
qb = QuantBook()
byte_data = qb.object_store.read_bytes("/market-regime-signals.zip")
import io
import zipfile
import pandas as pd
with zipfile.ZipFile(io.BytesIO(byte_data)) as archive:
filename = archive.namelist()[0]
with archive.open(filename) as file:
df = pd.read_csv(file)
Live/backtest split: branch in pyget_sourcecsGetSource only when the source differs; return identical parsed objects from both paths.
Arrays/unfolding: keep the requested JSON array shape. Do not convert it to JSONL or CSV unless the user explicitly asks. Store Object Store JSON array files as one-line/minified JSON under the requested key, and do not change Object Store keys during debugging unless you report the change. Use pyFileFormat.UNFOLDING_COLLECTIONcsFileFormat.UnfoldingCollection, parse the array, sort emitted objects by pyend_timecsEndTime, and return pyBaseDataCollection(objects[-1].end_time, config.symbol, objects)csnew BaseDataCollection(objects.Last().EndTime, config.Symbol, objects).
using Newtonsoft.Json.Linq;
using System.Collections.Generic;
using System.Globalization;
using System.Linq;
publicoverride SubscriptionDataSource GetSource(SubscriptionDataConfig config, DateTime date, bool isLiveMode)
{
returnnew SubscriptionDataSource(
"custom-data/custom-news-releases.json",
SubscriptionTransportMedium.ObjectStore,
FileFormat.UnfoldingCollection);
}
publicoverride BaseData Reader(SubscriptionDataConfig config, string line, DateTime date, bool isLiveMode)
{
var objects = new List<BaseData>();
foreach (var row in JArray.Parse(line))
{
var data = new MyCustomData { Symbol = config.Symbol };
data.Time = DateTime.ParseExact(row.Value<string>("date"), "yyyy-MM-dd", CultureInfo.InvariantCulture);
data.EndTime = data.Time.AddDays(1);
data.Headline = row.Value<string>("headline");
data.Impact = row.Value<decimal>("impact");
data.Value = data.Impact;
objects.Add(data);
}
objects = objects.OrderBy(x => x.EndTime).ToList();
return objects.Count > 0 ? new BaseDataCollection(objects.Last().EndTime, config.Symbol, objects) : null;
}
Universes: emit symbols, log selected count at each rebalance, and skip the single-symbol history check.
6. Compile and backtest loop
Compile first; fix every build error before backtesting.
Backtest the smallest date window that covers one representative record. Use the full file only for date-dependent file selection, unfolding behavior, or live/backtest branching.
Confirm History rows: N with N > 0 for non-universe readers.
If N == 0, inspect the first real record and source path, then manually walk date parsing, transport medium, resolution, and start/end dates.
Preserve the user's trading rule. If data loads but no orders appear, log/report condition pass counts and explain whether the provided sample data satisfies the rule. Only relax strategy logic if the user explicitly asks for a smoke-test trade.
Before finishing, verify or mark not applicable: linked, unlinked, universe, unfolding collection, ZIP, remote file, Object Store, and target language.
Report compile result, backtest result, loaded row count, order count, and Object Store key or remote URL.
