Tos File Access

TOS File Access

This skill provides utilities for uploading files and directories to TOS-compatible object storage for Volcano Engine or BytePlus, and downloading files from URLs.

Overview

TOS-compatible object storage is used by Volcano Engine and BytePlus, similar to AWS S3. This skill enables:

• Upload: Upload Agent-generated files or entire directories to TOS and get shareable signed URLs (for files) or TOS paths (for directories)
• Download: Download files from URLs to local storage for Agent processing

Typical Workflows

Pre-Agent Execution: Download User Files

When users provide file URLs (TOS or external), download them before processing:

# Download single file
python scripts/file_download.py https://example.com/data.csv

# Download multiple files
python scripts/file_download.py https://example.com/data.csv https://example.com/config.json

# Specify save directory and filenames
python scripts/file_download.py https://example.com/data.csv --save-dir /workspace --filenames dataset.csv

Post-Agent Execution: Upload Output Files or Directories

After generating files or directories (videos, charts, reports, output folders, etc.), upload them to TOS for user access:

# Upload single file (auto-detected)
python scripts/tos_upload.py /path/to/output.mp4 --bucket my-bucket

# Upload entire directory (auto-detected)
python scripts/tos_upload.py /path/to/output_folder --bucket my-bucket

# Upload to BytePlus with custom expiration
TOS_REGION=ap-southeast-1 python scripts/tos_upload.py /path/to/report.pdf --bucket my-bucket --expires 86400

Scripts

scripts/file_download.py

Download files from URLs to local storage.

Usage:

python scripts/file_download.py <url1> [url2 ...] [--save-dir DIR] [--filenames NAME1 NAME2 ...]

Arguments:

• urls: One or more URLs to download (positional, required)
• --save-dir: Save directory (optional, defaults to /tmp)
• --filenames: Custom filenames for downloaded files (optional, must match number of URLs)

Examples:

# Download single file to /tmp
python scripts/file_download.py https://tos-cn-beijing.volces.com/bucket/file.pdf

# Download to specific directory
python scripts/file_download.py https://example.com/data.json --save-dir /workspace/data

# Download multiple files with custom names
python scripts/file_download.py \
  https://example.com/file1.pdf \
  https://example.com/file2.jpg \
  --save-dir /workspace \
  --filenames document.pdf image.jpg

Returns: Prints absolute paths of downloaded files (one per line)

scripts/tos_upload.py

Upload files or directories to TOS and generate signed access URLs (for files) or TOS paths (for directories).

Key Features:

• Auto-detection: Automatically detects whether the path is a file or directory
• Session-based paths: Uses TOOL_USER_SESSION_ID environment variable to organize uploads
• Preserves structure: For directories, maintains the full directory structure in TOS
• Automatic bucket creation: Creates bucket if it doesn't exist (with private ACL)

Usage:

python scripts/tos_upload.py <path> --bucket BUCKET [--region REGION] [--expires SECONDS]

Arguments:

• path: Local file or directory path to upload (positional, required)
• --bucket: TOS bucket name (required)
• --region: TOS region (optional, defaults to TOS_REGION, DATABASE_TOS_REGION, or provider default: ap-southeast-1 for BytePlus, cn-beijing for Volcano Engine)
• --expires: Signed URL expiration in seconds (optional, defaults to 604800 = 7 days, only applies to file uploads)

Upload Structure:

• File: upload/{session_prefix}/{filename}
  • Example: upload/skill_agent_veadk_default_user_tmp-session-20251210150057/video.mp4
• Directory: upload/{session_prefix}/{directory_name}/{relative_path}
  • Example: upload/skill_agent_veadk_default_user_tmp-session-20251210150057/output_folder/file1.txt

Session Prefix:

• If TOOL_USER_SESSION_ID is set, uses that value as prefix
• Otherwise, falls back to timestamp format YYYYMMDD_HHMMSS

Cloud provider and authentication: Requires one of:

• TOS_REGION=ap-southeast-1 selects BytePlus endpoint first
• CLOUD_PROVIDER=byteplus selects BytePlus endpoint when TOS_REGION is not ap-southeast-1
• Environment variables: VOLCENGINE_ACCESS_KEY and VOLCENGINE_SECRET_KEY
• VeFaaS IAM Role (automatic credential retrieval)

Examples:

# Upload single file (auto-detected)
python scripts/tos_upload.py /workspace/output.mp4 --bucket my-bucket

# Upload entire directory (auto-detected)
python scripts/tos_upload.py /workspace/results_folder --bucket my-bucket

# Upload to BytePlus with 1-day expiration
TOS_REGION=ap-southeast-1 python scripts/tos_upload.py /workspace/report.pdf \
  --bucket my-reports \
  --expires 86400

# Upload directory with all options
python scripts/tos_upload.py /workspace/output_dir \
  --bucket data-storage \
  --region cn-beijing

Returns:

• For files: Prints a signed URL that can be shared with users (valid for specified duration)
• For directories: Prints a TOS path in format tos://bucket-name/path/to/directory

Output Examples:

# File upload output
============================================================
✅ Upload Successful!
============================================================
Signed URL:
https://my-bucket.tos-cn-beijing.volces.com/upload/skill_agent_xxx/video.mp4?X-Tos-Signature=...
============================================================

# Directory upload output
============================================================
✅ Upload Successful!
============================================================
TOS Path:
tos://my-bucket/upload/skill_agent_xxx/output_folder
============================================================

Environment Variables

• TOS_REGION: TOS region (optional). ap-southeast-1 selects BytePlus endpoint.
• CLOUD_PROVIDER: Cloud provider (optional). byteplus selects BytePlus endpoint when TOS_REGION is not ap-southeast-1; otherwise Volcano Engine endpoint is used.
• VOLCENGINE_ACCESS_KEY: Access key for object storage authentication
• VOLCENGINE_SECRET_KEY: Secret key for object storage authentication
• DATABASE_TOS_REGION: Optional region fallback when --region and TOS_REGION are not provided
• TOOL_USER_SESSION_ID: Session ID used to generate organized upload paths (optional, falls back to timestamp)

Common Use Cases

1. Video Processing: Download source video → process → upload result

   # Download
   python scripts/file_download.py https://example.com/input.mp4 --save-dir /workspace
   # Process video (your code here)
   # Upload result
   python scripts/tos_upload.py /workspace/output.mp4 --bucket video-storage
   

2. Data Analysis: Download datasets → analyze → upload charts/reports folder

   # Download data
   python scripts/file_download.py https://example.com/data.csv --save-dir /workspace
   # Analyze and generate charts (your code here)
   # Upload entire results folder
   python scripts/tos_upload.py /workspace/analysis_results --bucket analysis-bucket
   

3. Batch Processing: Process multiple files and upload output directory

   # Process files (your code here, generates /workspace/batch_output/)
   # Upload entire output directory
   python scripts/tos_upload.py /workspace/batch_output --bucket batch-results
   

4. Project Export: Upload a complete project directory structure

   # Upload project directory with subdirectories
   python scripts/tos_upload.py /workspace/my_project --bucket project-storage
   # Result: tos://project-storage/upload/{session_id}/my_project/...
   

Error Handling

• IF the script raises the error VOLCENGINE_ACCESS_KEY and VOLCENGINE_SECRET_KEY are not provided or IAM Role is not configured., inform the user that they need to provide the VOLCENGINE_ACCESS_KEY and VOLCENGINE_SECRET_KEY environment variable. Write it to the environment variable file in the workspace. If the file already exists, append it to the end. Ensure the environment variable format is correct, make the environment variable effective, and retry the task that just failed.

Notes

• Auto-detection: No need to specify whether uploading a file or directory - the script detects automatically
• Session organization: Files and directories are automatically organized by session ID for easy tracking
• Signed URLs (files only): Valid for 7 days by default (adjustable via --expires)
• TOS Paths (directories): Returned as tos://bucket/path format for reference
• Structure preservation: Directory uploads maintain the complete folder structure in TOS
• No timestamp in filenames: Original filenames are preserved (session prefix provides uniqueness)
• Auto-bucket creation: Bucket is automatically created with private ACL if it doesn't exist
• Auto-deduplication: Downloads automatically rename files if they already exist
• IAM Role support: Scripts automatically retrieve credentials from VeFaaS IAM when available
• Error handling: Scripts print clear error messages for network, permission, or file issues
• Bucket requirement: Bucket name must be specified via --bucket parameter (required)
• Endpoint selection: TOS_REGION=ap-southeast-1 takes priority for BytePlus. Otherwise CLOUD_PROVIDER=byteplus selects BytePlus. All other cases use Volcano Engine.