stacks-storage

# Stacks Storage File system abstraction with a Laravel-style Storage facade, local/S3/Azure adapters, file upload handling, and low-level file utilities. ## Key Paths - Core package: `storage/framework/core/storage/src/` - Storage facade: `storage/framework/core/storage/src/facade.ts` - Uploaded file: `storage/framework/core/storage/src/uploaded-file.ts` - Types: `storage/framework/core/storage/src/types.ts` - Filesystem config types: `storage/framework/core/storage/src/types/filesystem.ts` - Local adapter: `storage/framework/core/storage/src/adapters/local.ts` - S3 adapter: `storage/framework/core/storage/src/adapters/s3.ts` - Azure adapter: `storage/framework/core/storage/src/adapters/azure.ts` - Azure request signing: `storage/framework/core/storage/src/azure-signing.ts` - Memory adapter: `storage/framework/core/storage/src/adapters/memory.ts` - Bun adapter: `storage/framework/core/storage/src/adapters/bun.ts` - File utilities: `storage/framework/core/storage/src/files.ts` - Bounded local text reads: `storage/framework/core/storage/src/bounded-text-file.ts` - Completed local artifacts: `storage/framework/core/storage/src/completed-artifact-directory.ts` - Folder utilities: `storage/framework/core/storage/src/folders.ts` - Copy: `storage/framework/core/storage/src/copy.ts` - Move: `storage/framework/core/storage/src/move.ts` - Delete: `storage/framework/core/storage/src/delete.ts` - Hash: `storage/framework/core/storage/src/hash.ts` - Glob: `storage/framework/core/storage/src/glob.ts` - Zip: `storage/framework/core/storage/src/zip.ts` - Helpers: `storage/framework/core/storage/src/helpers.ts` - Configuration: `config/filesystems.ts` ## Package Exports ```typescript // Storage facade (Laravel-style) import { Storage, StorageManager } from '@stacksjs/storage' // File uploads import { UploadedFile, uploadedFile, uploadedFiles } from '@stacksjs/storage' // Adapters import { createLocalStorage, LocalStorageAdapter } from '@stacksjs/storage' import { createS3Storage, S3StorageAdapter } from '@stacksjs/storage' // Config helpers import { localDisk, s3Disk, azureDisk, r2Disk, gcsDisk, filebaseDisk, backblazeDisk, hetznerDisk, configFromEnv } from '@stacksjs/storage' // Types import type { StorageAdapter, FileContents, StatEntry, DirectoryEntry, DirectoryListing } from '@stacksjs/storage' import type { ListOptions, PublicUrlOptions, TemporaryUrlOptions, ChecksumOptions, MimeTypeOptions } from '@stacksjs/storage' import type { DiskConfig, FilesystemConfig, LocalDiskConfig, S3DiskConfig } from '@stacksjs/storage' import { Visibility, createDirectoryListing, normalizeExpiryToMilliseconds, normalizeExpiryToDate, isFile, isDirectory } from '@stacksjs/storage' // Low-level file operations import { copy, copyFile, copyFolder } from '@stacksjs/storage' import { storage } from '@stacksjs/storage' // storage.move, storage.rename import { del, deleteFile, deleteFolder, deleteEmptyFolder, deleteEmptyFolders, deleteGlob, isDirectoryEmpty } from '@stacksjs/storage' import { readJsonFile, readPackageJson, readTextFile, writeFile, writeJsonFile, writeTextFile, put, get, getFiles, deleteFiles, hasFiles } from '@stacksjs/storage' import { isFolder, isDir, doesFolderExist, createFolder, getFolders } from '@stacksjs/storage' import { glob, globSync } from '@stacksjs/storage' import { hashDirectory, hashPath, hashPaths } from '@stacksjs/storage' import { zip, unzip, archive, unarchive, compress, decompress, gzipSync, gunzipSync, deflateSync, inflateSync, zipEntries } from '@stacksjs/storage' import type { ZipEntriesOptions, ZipEntry } from '@stacksjs/storage' import * as storage from '@stacksjs/storage' ``` ## Storage Facade (StorageManager) ### Completed local artifacts Use `writeCompletedArtifact` from `@stacksjs/storage/completed-artifact-directory` for a bounded, new local output directory containing data files and a JSON completion manifest. Pass `(directory, [{ path, contents }], manifestJson)`; optional `{ directories: ['images'] }` creates empty folders. It snapshots bytes, validates paths/collisions/budgets, writes private exclusive files, syncs files and directories, then hard-links the synced manifest to its completion name. Read the package README for exact limits and failure behavior. Callers own parent-directory authorization and domain validation. This is a local hard-link and directory-fsync protocol, not a cloud-adapter transaction or tested hardware power-loss guarantee. For a durable ownership checkpoint before private bytes, use `prepareCompletedArtifactDirectorySync(directory, entries, manifestJson, options)` from the same subpath or package root. Its deeply frozen handle contains only `receipt` and single-use `publishSync()`. Preparation exclusively creates and syncs a private empty tree, then closes every descriptor. Persist the receipt before calling the same live handle's publication method. The receipt records canonical parent/tree/file identities and planned lengths/SHA-256; observed file lengths are zero and `manifest.json` is absent. Planned metadata is not completed content or a deletion receipt. Pass plain data records/lists and unshared Uint8Array/Buffer views. Record/list validation refuses proxies, getters, holes and hidden/symbol aliases without executing caller code; byte views use intrinsic offsets rather than overrides. Publication rechecks the empty tree and identities, writes/syncs detached bytes, verifies readback hashes and exclusively links the registered pending-manifest inode to completion. Every attempt consumes the handle, even on error. Errors preserve state; JSON cannot rehydrate a writer. There is no overwrite, arbitrary existing-directory writer or implicit cleanup. Read the storage README's prepared-artifact section before integrating an ownership ledger. It defines decimal-string identities, planned byte semantics, birth-time availability, prepublication-only change time and legitimate link transitions. Parent paths must already exist, belong to the operator and remain controlled. Canonical ancestor aliases are allowed; final targets are exclusive. Both calls block the isolate, including while holding any caller transaction. Caps bound bytes/files rather than I/O duration; database contention and SQL/FS atomicity remain caller concerns. The existing async writer keeps asynchronous filesystem execution. Real process-interruption tests do not establish hardware power-loss, hostile same-user race, deletion or media-erasure guarantees. ### Recoverable temporary workspaces Use `TemporaryWorkspaceJournal` for workers that recover temporary directories after a process interruption. Keep its private journal outside job-controlled checkouts and separate it by stable worker and controller identity. `create(key)` returns a frozen live ownership handle; stop its resources, then call `complete(handle)` for normal disposal. A restarted instance cannot forge this capability from a listed record. Both live completion and recovered retirement recheck the current mount table immediately before deletion. Nested mounts, including bind mounts on the same device, preserve the workspace and journal. Incomplete mount inspection also preserves them: recursive deletion can unlink mounted data before reporting that the mount point is busy. Crash recovery uses `inspectRetirement(id, mayRetire)` for a read-only preview and `retire(id, mayRetire)` to apply the same checks. Both require an authoritative lifecycle decision and preserve changed, mounted or in-use directories. Age alone never authorizes deletion. `inspectTemporaryWorkspace` reports unavailable process visibility explicitly. Protected same-user peers can require privileged maintenance on Linux; never ignore permission errors. An already-privileged operator may set `ownerUid` to maintain an existing private journal belonging to that user. This does not create another user's workspaces or grant new privileges. Unsupported or incomplete inspection always preserves data. ```ts import { TemporaryWorkspaceJournal } from '@stacksjs/storage' const journal = new TemporaryWorkspaceJournal({ directory: privateJournalDirectory, temporaryRoot, prefix: 'worker-job-' }) const workspace = journal.create(jobId) // Stop the live workspace's resources before normal disposal. journal.complete(workspace) // After restart, ask the controller about a recovered record. await journal.retire(recovered.id, record => controllerConfirmsFinished(record.key)) ``` The `Storage` singleton is a pre-instantiated `StorageManager`. It lazily builds its config from `@stacksjs/config` (which reads `config/filesystems.ts` and env vars). ### Directory mounts without retirement Use `inspectDirectoryMounts(canonicalDirectory)` for read-only current-namespace mount checks where legitimate readers may stay open. Supply an existing, checked canonical directory. An unmounted result means only that the native mount table did not reference that directory or a descendant. Unavailable, unsupported, aliased or missing paths never return `unmounted: true`. Keep `inspectTemporaryWorkspace` and journal retirement for deletion: they also check process references and inspection permissions. Mount inspection alone grants no ownership, quiescence or retirement authority and does not inspect other namespaces or prevent later host changes. ### Basic Operations ```typescript import { Storage } from '@stacksjs/storage' // Write a file (to default disk) await Storage.put('file.txt', 'Hello World') await Storage.put('data.bin', new Uint8Array([1, 2, 3])) // Read a file as string const content = await Storage.get('file.txt') // Check existence const exists = await Storage.exists('file.txt') const missing = await Storage.missing('file.txt') // Delete await Storage.delete('file.txt') // Copy and move await Storage.copy('source.txt', 'dest.txt') await Storage.move('old.txt', 'new.txt') // File info const size = await Storage.size('file.txt') // bytes const modified = await Storage.lastModified('file.txt') // ms timestamp const mime = await Storage.mimeType('file.txt') const hash = await Storage.checksum('file.txt', 'sha256') // 'md5' | 'sha1' | 'sha256' // Public URL const url = await Storage.url('file.txt') // Directories await Storage.makeDirectory('uploads/images') await Storage.deleteDirectory('uploads/old') // List files (returns AsyncIterable<{ path, type }>) for await (const entry of Storage.files('uploads/')) { console.log(entry.path, entry.type) // 'file' or 'directory' } // Recursive listing for await (const entry of Storage.allFiles('uploads/')) { console.log(entry.path) } ``` ### Using Named Disks ```typescript // Use a specific disk await Storage.disk('s3').write('uploads/file.txt', contents) await Storage.disk('public').write('images/logo.png', imageData) await Storage.disk('local').readToString('config.json') // Disk instances are cached -- subsequent calls return the same adapter ``` ### Configuring Disks ```typescript // Initialize with custom config (overrides env-based config) Storage.init({ default: 's3', disks: { custom: { driver: 'local', root: '/custom/path' }, }, }) // Add/update a disk at runtime Storage.configure('backups', { driver: 's3', bucket: 'my-backups', region: 'eu-west-1' }) // Change default disk Storage.setDefaultDisk('s3') // Query configuration Storage.getDefaultDisk() // 'local' Storage.getConfiguredDisks() // ['local', 'public', 's3'] Storage.getDiskConfig('s3') // S3DiskConfig object // Reset all caches (useful for testing) Storage.reset() ``` ### Built-in Disk Configurations The facade auto-configures these disks from `config/filesystems.ts`: - **`local`** -- `driver: 'local'`, root: `<project>/storage/app`, visibility: from config (default `'private'`) - **`public`** -- `driver: 'local'`, root: `<project>/public`, url: the application origin, visibility: `'public'` - **`s3`** -- only added if `s3.bucket` is configured in filesystems config ## StorageAdapter Interface All adapters implement this interface: ```typescript interface StorageAdapter { write(path: string, contents: FileContents): Promise<PutResult> read(path: string): Promise<FileContents> readToString(path: string): Promise<string> readToBuffer(path: string): Promise<Buffer> readToUint8Array(path: string): Promise<Uint8Array> deleteFile(path: string): Promise<void> deleteDirectory(path: string): Promise<void> createDirectory(path: string): Promise<void> moveFile(from: string, to: string): Promise<void> copyFile(from: string, to: string): Promise<void> stat(path: string): Promise<StatEntry> list(path: string, options?: ListOptions): DirectoryListing // AsyncIterable changeVisibility(path: string, visibility: Visibility): Promise<void> visibility(path: string): Promise<Visibility> fileExists(path: string): Promise<boolean> directoryExists(path: string): Promise<boolean> publicUrl(path: string, options?: PublicUrlOptions): Promise<string> temporaryUrl(path: string, options: TemporaryUrlOptions): Promise<string> checksum(path: string, options?: ChecksumOptions): Promise<string> mimeType(path: string, options?: MimeTypeOptions): Promise<string> lastModified(path: string): Promise<number> fileSize(path: string): Promise<number> } ``` ### Local Adapter (`LocalStorageAdapter`) ```typescript import { createLocalStorage } from '@stacksjs/storage' const local = createLocalStorage({ root: './storage' }) ``` - Uses Node.js `fs/promises` for file operations - Path traversal protection: throws if resolved path escapes the root directory - `write()` auto-creates parent directories with `mkdir({ recursive: true })` - `moveFile()` uses `fs.rename()`; `copyFile()` uses `fs.copyFile()` - `checksum()` uses `Bun.CryptoHasher` (default algorithm: `sha256`) - `temporaryUrl()` generates HMAC-signed URLs using `APP_KEY` env var - `changeVisibility()` changes POSIX modes: public files/directories use 0644/0755, private use 0600/0700 - `visibility()` reads mode bits; a world-readable entry is public, other modes collapse to private - MIME type detection is extension-based (supports txt, html, css, js, json, xml, pdf, zip, jpg, jpeg, png, gif, svg, mp4, mp3, wav) - `list()` returns an async iterable; supports `deep: true` for recursive listing ### S3 Adapter (`S3StorageAdapter`) ```typescript import { createS3Storage, S3StorageAdapter } from '@stacksjs/storage' import { S3Client } from '@stacksjs/ts-cloud' const client = new S3Client('us-east-1') const s3 = new S3StorageAdapter(client, { bucket: 'my-bucket', region: 'us-east-1', prefix: 'uploads/' }) ``` - Uses `@stacksjs/ts-cloud` S3Client for AWS operations - Supports key prefix: all paths are prefixed with `config.prefix` - `write()` auto-detects MIME type from extension and sets `contentType` - `createDirectory()` is a no-op (S3 directories are implicit) - `moveFile()` = `copyFile()` + `deleteFile()` - `deleteDirectory()` lists all objects with prefix and deletes them in bulk - `temporaryUrl()` uses `client.getSignedUrl()` for pre-signed URLs - `checksum()` downloads the file content and hashes with `Bun.CryptoHasher` - `publicUrl()` defaults to `https://<bucket>.s3.<region>.amazonaws.com/<key>` - `list()` supports pagination via continuation tokens; `deep: true` uses `listAllObjects()` - `fileExists()` uses `headObject()` and catches 404/NoSuchKey/NotFound errors Every S3-COMPATIBLE provider is an `s3` disk with a different endpoint, and has a preset that fills it in: `r2Disk(bucket, accountId)`, `gcsDisk(bucket)`,