battery-data-preprocess

name	battery-data-preprocess
description	Preprocess battery cycling data from raw formats (parquet/csv/xlsx/mat/pkl/json) into standardized pkl format. Includes cycle detection, capacity calculation, outlier removal, time normalization, RUL computation, and visualization. Use when working with battery datasets, cycling data, capacity degradation, or battery life prediction.

Battery Data Preprocessing

Overview

将电池原始循环数据转换为标准化 pkl 格式，支持多种输入格式和数据集。

Core Data Structures

CycleData (per-cycle dict)

{
    'cycle_number': int,
    'voltage_in_V': list or np.float32 array,
    'current_in_A': list or np.float32 array,
    'charge_capacity_in_Ah': list or np.float32 array,     # cumulative
    'discharge_capacity_in_Ah': list or np.float32 array,  # cumulative
    'time_in_s': list or np.float32 array,                 # cumulative across cycles
    'temperature_in_C': list or np.float32 array or None,
    'internal_resistance_in_ohm': float or None,
}

BatteryData (top-level dict)

{
    'cell_id': str,
    'cycle_data': [CycleData, ...],
    'form_factor': str or None,         # 'cylindrical_18650', 'prismatic', 'pouch'
    'anode_material': str or None,
    'cathode_material': str or None,
    'electrolyte_material': str or None,
    'nominal_capacity_in_Ah': float,
    'depth_of_charge': 1.0,
    'depth_of_discharge': 1.0,
    'already_spent_cycles': 0,
    'max_voltage_limit_in_V': float,
    'min_voltage_limit_in_V': float,
    'max_current_limit_in_A': float or None,
    'min_current_limit_in_A': float or None,
    'reference': str or None,
    'description': str or None,
    'charge_protocol': [ProtocolDict, ...],
    'discharge_protocol': [ProtocolDict, ...],
    'SOC_interval': [0, 1],
}

ProtocolDict

{
    'rate_in_C': float or str,
    'current_in_A': float or None,
    'voltage_in_V': float or None,
    'power_in_W': float or None,
    'start_voltage_in_V': float or None,
    'start_soc': float,
    'end_voltage_in_V': float or None,
    'end_soc': float,
}

Workflow

- [ ] Step 1: Load raw data & identify columns
- [ ] Step 2: Separate RPT data (if applicable)
- [ ] Step 3: Detect cycle boundaries
- [ ] Step 4: Extract per-cycle data (V, I, T, capacity)
- [ ] Step 5: Data cleaning (remove anomalous cycles)
- [ ] Step 6: Time normalization
- [ ] Step 7: Build standardized pkl
- [ ] Step 8: Outlier removal & smoothing (degradation analysis)
- [ ] Step 9: RUL calculation
- [ ] Step 10: Visualization

Step 1: Load & Identify Columns

Supported formats: parquet, csv, xlsx, mat (h5py), pkl, json.

Auto-detect columns by matching known patterns:

Role	Known Column Names
timestamp	time_stamp, Time, time, Test_Time (s), Test_Time(s), test_time, time/s
current	current (A), Current(A), Current (A), <I>/mA (÷1000), Current (mA) (÷1000), I
voltage	voltage (V), Voltage(V), Voltage (V), Ecell/V, V
temperature	temp, Temperature, Cell_Temperature (C), temperature
cycle	cycle number, Cycle_Index, cycle_index, cycle
charge_cap	Q charge/mA.h (÷1000), Charge_Capacity (Ah), charge_capacity, Qc
discharge_cap	Q discharge/mA.h (÷1000), Discharge_Capacity (Ah), discharge_capacity, Qd
rpt	rpt, RPT, test_type

Step 2: RPT Separation

if rpt_column exists:
    rpt_data = df[df[rpt_col].notna()]    # preserve separately
    cycling = df[df[rpt_col].isna()]       # use for cycle extraction

Step 3: Cycle Detection

Strategy A (preferred): Explicit cycle column → groupby('cycle')

Strategy B: Current sign change (for datasets without cycle column):

THRESH = 5.0  # adjust per dataset
# Detect transition from discharge(I<0) to charge(I>0)

Strategy C: SOC valleys (when current signal is weak)

Step 4: Per-Cycle Extraction

If raw data has capacity columns, use directly (with unit conversion). Otherwise, compute from current integration:

# Vectorized
avg_curr = (current[1:] + current[:-1]) / 2.0
dq = avg_curr * dt[1:] / 3600.0
charge_cap[1:] = np.cumsum(np.where(avg_curr > 0, dq, 0))
discharge_cap[1:] = np.cumsum(np.where(avg_curr < 0, -dq, 0))

Step 5: Data Cleaning

A. Capacity threshold: Remove cycles with Qd below a dataset-specific minimum.

B. Median + Hampel filter:

from scipy.signal import medfilt
Qd_med = medfilt(Qd, 21)
ths = np.median(abs(Qd - Qd_med))
should_keep = abs(Qd - Qd_med) < 3 * ths

C. Left-right deviation + Hampel:

diff_left = abs(Qd - np.roll(Qd, shift))
diff_right = abs(Qd - np.roll(Qd, -shift))
diff = np.amin([diff_left, diff_right], 0)
should_exclude = (abs(diff - np.median(diff)) > np.median(abs(diff - np.median(diff))) * 3)

Step 6: Time Normalization

Ensure cumulative, continuous time across all cycles:

• Fix internal resets (time goes to 0 mid-cycle → accumulate offset)
• Compress large intra-cycle gaps (>5min → 1s)
• Handle special units (ns→s, ms→s)
• Make each cycle's time relative, then add cumulative offset

Step 7: Build PKL

Use np.float32 arrays for compact storage. Save with pickle.dump(result, f, protocol=4).

Step 8: Outlier Removal & Smoothing

For post-hoc degradation analysis:

# Iterative local median filter (3 passes, window=15)
# Threshold: max(2.5 × local_std, 1.5 Ah)
# Then Savitzky-Golay: window=51, polyorder=3

Step 9: RUL Calculation

threshold = initial_capacity * 0.9
below = np.where(caps_smooth < threshold)[0]
rul = int(cycles[below[0]]) if len(below) > 0 else None

Step 10: Visualization

1. Capacity degradation: raw scatter + smoothed line + 90% threshold + RUL marker
2. Per-cycle V-I: voltage & current vs time (twin axes)
3. Multi-cycle overlay: N cycles overlaid
4. Discharge curves: voltage vs discharge capacity

Dataset Fingerprints

Column Signature	Dataset	Key Notes
Ecell/V, <I>/mA, Q charge/mA.h, cycle number	Tongji	mA/mAh units, skip cycle<2
time_stamp, current (A), voltage (V), rpt, soc	NUS	Separate RPT, current sign detection
Cycle_Index, Current(A), Voltage(V), Test_Time(s)	CALCE	Median filter cleaning
Cycle_Index, Current (A), Voltage (V), Test_Time (s)	HNEI	Skip first 12 cycles, imputation
cycle_index, current, voltage, temperature	Stanford	JSON format, skip formation
Current (mA), Voltage (V), Time (s)	HUST	pkl format, mA→A
.mat with batch/cycles/I,V,t,Qc,Qd	MATR	h5py, time in minutes→s

Common Unit Conversions

From	To	Operation
mA	A	÷1000
mV	V	÷1000
mAh	Ah	÷1000
min	s	×60
ns	s	÷1e9
ms	s	÷1000

Standalone Script

python scripts/battery_preprocess.py \
  --input_dir /path/to/raw \
  --output_dir /path/to/output \
  --cell_prefix "DATASET" \
  --visualize

See reference.md for full CLI parameters. See examples.md for usage examples.