megfile

name	megfile
description	Use megfile’s APIs, SmartPath, configuration, and CLI to perform filesystem tasks easy across local FS, S3/OSS-style object storage, SFTP, WebDAV, HTTP, HDFS, and stdio.

Megfile

Overview

Guide to using megfile’s unified APIs, SmartPath, configuration, and CLI for file operations across various backends, like local FS, S3/OSS-compatible object storage, SFTP, WebDAV, HTTP, HDFS, and stdio.

Quick Start

• Install base: pip install megfile; add extras per backend (megfile[cli], megfile[hdfs], megfile[webdav]).
• Configure credentials/endpoints (env vars > config files). S3 examples: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, optional AWS_ENDPOINT_URL / OSS_ENDPOINT / AWS_ENDPOINT_URL_S3, AWS_S3_ADDRESSING_STYLE.
• Path format: protocol://bucket/key or alias (e.g., tos://); bare POSIX paths are treated as file://.
• Import functional APIs (from megfile import smart_open, smart_sync, ...) or SmartPath (from megfile.smart_path import SmartPath).

Supported Protocols & Extras

• Local FS (file:// or bare paths) — base install.
• S3/OSS-compatible (s3://, plus aliases) — base install.
• SFTP (sftp://) — install megfile[cli] or megfile with SFTP deps.
• HTTP/HTTPS (http://, https://) — base install.
• Stdio (stdio://) — base install.
• HDFS (hdfs://) — install megfile[hdfs].
• WebDAV (webdav://) — install megfile[webdav].
• Full Protocol path format reference: references/path_format.md.

Core Tasks

File IO

• smart_open(path, mode='r', encoding=None, **options): open binary/text handles.
• Convenience loaders/savers: smart_load_content / smart_save_content (bytes), smart_load_text / smart_save_text (str), smart_save_as(file_obj, path), smart_load_from(path) returns BinaryIO.
• smart_combine_open(glob, mode='rb', open_func=smart_open): sequentially reads multiple files as one stream.

Existence & Metadata

• smart_exists, smart_isfile, smart_isdir, smart_islink, smart_isabs.
• smart_access(path, mode=Access.READ/WRITE).
• smart_stat / smart_lstat (via SmartPath), smart_getsize, smart_getmtime, smart_getmd5(recalculate=False, followlinks=False).

Listing & Globbing

• Directory traversal: smart_listdir, smart_scandir, smart_walk, smart_scan, smart_scan_stat (returns FileEntry/StatResult).
• Pattern matching: smart_glob, smart_iglob, smart_glob_stat for stat-rich globbing.

Data Transfer & Lifecycle

• Copy/sync: smart_copy, smart_sync, smart_sync_with_progress (progress-friendly wrapper), smart_concat to merge multiple sources.
• Moves/deletes: smart_move, smart_rename, smart_remove, smart_unlink, smart_touch, smart_makedirs.
• Links: smart_symlink, smart_readlink.

Path Utilities

• smart_path_join, smart_abspath, smart_realpath, smart_relpath, smart_isabs.
• SmartPath mirrors pathlib semantics but routes to the right backend: path = SmartPath("s3://bucket/key"); path.exists(); path.open(mode="rb").

Caching

• smart_cache(path, cacher=SmartCacher, **options): cache remote resources locally for tools that only support local files.

Configuration

• Use CLI helpers to persist credentials/endpoints; environment variables take precedence.
• Profiles enable multiple endpoints (e.g., s3+prod://...). See references/configuration/ for protocol-specific flags and env vars.
• Full config reference: references/configuration/.

CLI Essentials

• Install CLI extras: pip install 'megfile[cli]'.
• Common commands (ls/cp/sync/stat/md5sum/mkdir/rm/touch) mirror POSIX semantics across backends.
• Completion scripts: megfile completion zsh.
• Full command list and flags: references/cli.md.

Usage Notes

• Prefer smart_* for protocol-agnostic code paths; avoid branching per backend.
• Ensure required extras are installed for target protocols before invoking APIs.
• For high-volume sync/copy, supply map_func (e.g., ThreadPoolExecutor.map) and callback to report progress.
• Use aliases via megfile config alias <alias> <protocol> to shorten paths (e.g., tos://).

References

• API surface: references/megfile.smart.md and references/megfile.smart_path.md.
• Configuration flags, env vars, and profiles: references/configuration/
• CLI commands and flags: references/cli.md
• Full Protocol path format reference: references/path_format.md.
• Glob patterns reference: references/advanced/glob.md.