# Megfile > Use megfile’s APIs, SmartPath, configuration, and CLI to perform filesystem tasks easy across local FS, S3/OSS-style object storage, SFTP, WebDAV, HTTP, HDFS, and stdio. Use when this capability is needed. - Skill: `tomevault-io/megfile` (Agent Skill, multi-file: 2 files) - Install (CLI): `npx skillmds@latest add tomevault-io/megfile` - Raw SKILL.md: https://api.skillmd.com/api/skills/tomevault-io/megfile/raw - Safety review: pending (external: skill-scanner PASS, skillspector PASS) - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Research & Search - Author: tomevault-io (https://skillmd.com/u/tomevault-io) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/tomevault-io/megfile --- # Megfile ## Overview Guide to using megfile’s unified APIs, SmartPath, configuration, and CLI for file operations across various backends, like local FS, S3/OSS-compatible object storage, SFTP, WebDAV, HTTP, HDFS, and stdio. ## Quick Start - Install base: `pip install megfile`; add extras per backend (`megfile[cli]`, `megfile[hdfs]`, `megfile[webdav]`). - Configure credentials/endpoints (env vars > config files). S3 examples: `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, optional `AWS_ENDPOINT_URL` / `OSS_ENDPOINT` / `AWS_ENDPOINT_URL_S3`, `AWS_S3_ADDRESSING_STYLE`. - Path format: `protocol://bucket/key` or alias (e.g., `tos://`); bare POSIX paths are treated as `file://`. - Import functional APIs (`from megfile import smart_open, smart_sync, ...`) or SmartPath (`from megfile.smart_path import SmartPath`). ## Supported Protocols & Extras - Local FS (`file://` or bare paths) — base install. - S3/OSS-compatible (`s3://`, plus aliases) — base install. - SFTP (`sftp://`) — install `megfile[cli]` or `megfile` with SFTP deps. - HTTP/HTTPS (`http://`, `https://`) — base install. - Stdio (`stdio://`) — base install. - HDFS (`hdfs://`) — install `megfile[hdfs]`. - WebDAV (`webdav://`) — install `megfile[webdav]`. - Full Protocol path format reference: `references/path_format.md`. ## Core Tasks ### File IO - `smart_open(path, mode='r', encoding=None, **options)`: open binary/text handles. - Convenience loaders/savers: `smart_load_content` / `smart_save_content` (bytes), `smart_load_text` / `smart_save_text` (str), `smart_save_as(file_obj, path)`, `smart_load_from(path)` returns BinaryIO. - `smart_combine_open(glob, mode='rb', open_func=smart_open)`: sequentially reads multiple files as one stream. ### Existence & Metadata - `smart_exists`, `smart_isfile`, `smart_isdir`, `smart_islink`, `smart_isabs`. - `smart_access(path, mode=Access.READ/WRITE)`. - `smart_stat` / `smart_lstat` (via SmartPath), `smart_getsize`, `smart_getmtime`, `smart_getmd5(recalculate=False, followlinks=False)`. ### Listing & Globbing - Directory traversal: `smart_listdir`, `smart_scandir`, `smart_walk`, `smart_scan`, `smart_scan_stat` (returns FileEntry/StatResult). - Pattern matching: `smart_glob`, `smart_iglob`, `smart_glob_stat` for stat-rich globbing. ### Data Transfer & Lifecycle - Copy/sync: `smart_copy`, `smart_sync`, `smart_sync_with_progress` (progress-friendly wrapper), `smart_concat` to merge multiple sources. - Moves/deletes: `smart_move`, `smart_rename`, `smart_remove`, `smart_unlink`, `smart_touch`, `smart_makedirs`. - Links: `smart_symlink`, `smart_readlink`. ### Path Utilities - `smart_path_join`, `smart_abspath`, `smart_realpath`, `smart_relpath`, `smart_isabs`. - SmartPath mirrors pathlib semantics but routes to the right backend: `path = SmartPath("s3://bucket/key"); path.exists(); path.open(mode="rb")`. ### Caching - `smart_cache(path, cacher=SmartCacher, **options)`: cache remote resources locally for tools that only support local files. ## Configuration - Use CLI helpers to persist credentials/endpoints; environment variables take precedence. - Profiles enable multiple endpoints (e.g., `s3+prod://...`). See `references/configuration/` for protocol-specific flags and env vars. - Full config reference: `references/configuration/`. ## CLI Essentials - Install CLI extras: `pip install 'megfile[cli]'`. - Common commands (ls/cp/sync/stat/md5sum/mkdir/rm/touch) mirror POSIX semantics across backends. - Completion scripts: `megfile completion zsh`. - Full command list and flags: `references/cli.md`. ## Usage Notes - Prefer smart_* for protocol-agnostic code paths; avoid branching per backend. - Ensure required extras are installed for target protocols before invoking APIs. - For high-volume sync/copy, supply `map_func` (e.g., `ThreadPoolExecutor.map`) and `callback` to report progress. - Use aliases via `megfile config alias ` to shorten paths (e.g., `tos://`). ## References - API surface: `references/megfile.smart.md` and `references/megfile.smart_path.md`. - Configuration flags, env vars, and profiles: `references/configuration/` - CLI commands and flags: `references/cli.md` - Full Protocol path format reference: `references/path_format.md`. - Glob patterns reference: `references/advanced/glob.md`. --- > Converted and distributed by [TomeVault](https://tomevault.io/claim/megvii-research) — claim your Tome and manage your conversions.