| name | octant-data-engine |
| description | Deep domain knowledge for Octant's N-dimensional data system, storage backends (Zarr, Icechunk), OctantBlock slicing, LRU caching (BlockCache), and background prefetching (BlockPrefetcher). Use when modifying or debugging src/data/. |
Octant Data Engine Skill
This skill guides development and debugging of Octant's N-dimensional tensor extraction, caching, and storage backends in src/data/.
Architecture Overview
Storage Backend (Zarr/Icechunk)
│
▼
BlockStore Trait (inspect, fetch_block, fetch_blocks)
│
▼
DatasetManager (StoreHandle registry & lifecycle)
│
▼
BlockPrefetcher (bounded background lookahead pool)
│
▼
BlockCache (LRU cache of OctantBlock hyperslabs)
│
▼
OctantBlock (N-D strided memory block)
│
▼
MatrixData / Volume (f32 renderable GPU payload)
Key Invariants & Best Practices
1. BlockStore Trait (src/data/block_store.rs)
To add a new storage backend:
- Implement
backend_name(&self) -> &'static str
- Implement
variables(&self) -> &[VariableInfo]
- Implement
inspect(&self) -> &DatasetMetadata
- Implement
fetch_block(&self, request: &BlockRequest) -> Result<OctantBlock, DataError>
- Implement
fetch_blocks(&self, requests: &[BlockRequest]) -> Result<Vec<OctantBlock>, DataError>
- Remote storage clients (
object_store::ClientOptions) must always configure explicit timeouts (.with_timeout(Duration::from_secs(30)) and .with_connect_timeout(Duration::from_secs(10))) to prevent worker starvation.
2. ()