Fastcopy File Transfer Optimization

FastCopy File Transfer Optimization

Skill by ara.so — Devtools Skills collection.

Overview

FastCopy is a high-performance file duplication and transfer utility designed for professionals who need fast, reliable bulk file operations. It features multi-threaded I/O engines, intelligent caching, automatic retry logic, and integrity verification using SHA-256 and CRC-32 checksums.

Key capabilities:

• Multi-threaded parallel transfers (up to 64 concurrent streams)
• YAML-based profile configuration for repeatable operations
• CLI and GUI modes for flexible integration
• Cross-platform support (Windows, macOS, Linux)
• Built-in integrity verification and auto-resume
• Scriptable automation with hooks and scheduling

Installation

Windows

# Download from official source
# Install using the provided installer or extract portable version
fastcopy.exe --version

macOS

# Using Homebrew (if available)
brew install fastcopy

# Or download DMG and install manually
fastcopy --version

Linux

# Download portable binary
curl -LO https://example.com/fastcopy-linux-x64
chmod +x fastcopy-linux-x64
sudo mv fastcopy-linux-x64 /usr/local/bin/fastcopy

# Verify installation
fastcopy --version

Basic Usage

Command Line Interface

Simple File Copy

# Basic syntax
fastcopy <source> <destination> [options]

# Copy single file
fastcopy /path/to/source.zip /backup/

# Copy directory recursively
fastcopy /data/projects/ /backup/projects/ --recursive

# Copy with maximum parallelism
fastcopy /media/raw/ /nas/archive/ --streams 64 --turbo

Common Options

# High priority transfer with debug logging
fastcopy source.iso /backups/ --priority high --log-level debug

# Skip integrity verification (faster but less safe)
fastcopy /large_dataset/ /backup/ --no-verify --streams 32

# Resume interrupted transfer
fastcopy /incomplete/ /destination/ --resume always

# Tag transfers for organization
fastcopy build.tar.gz /releases/ --tag "v2.3.1-production"

Profile-Based Configuration

YAML Profile Structure

Create reusable transfer profiles for consistent operations:

# production-backup.yaml
transfer:
  mode: "turbo"                # standard, balanced, turbo
  streams: 32                  # concurrent streams (1-64)
  buffer_size_mb: 256          # memory buffer per stream
  verify: true                 # enable SHA-256 + CRC-32 checks
  resume: always               # always, on_failure, never

paths:
  source: "/var/www/production"
  destination: "/mnt/backup/www"

filters:
  include:
    - "*.php"
    - "*.html"
    - "*.css"
    - "*.js"
  exclude:
    - "cache/*"
    - "temp/*"
    - ".git/*"

schedule:
  type: "cron"                 # one_time, watch, cron
  expression: "0 2 * * *"      # daily at 2 AM

hooks:
  on_start: "echo 'Starting backup...'"
  on_complete: "/opt/scripts/notify-success.sh"
  on_error: "/opt/scripts/alert-admin.sh"

logging:
  level: "info"                # debug, info, warning, error
  file: "/var/log/fastcopy/backup.log"
  max_size_mb: 100

Using Profiles

# Execute with profile
fastcopy --profile production-backup.yaml

# Override profile settings
fastcopy --profile backup.yaml --streams 48 --priority realtime

# Validate profile without executing
fastcopy --profile backup.yaml --dry-run --validate

Advanced Patterns

Automated Backup Script

#!/bin/bash
# daily-backup.sh

# Environment variables
BACKUP_SOURCE="${BACKUP_SOURCE:-/data/production}"
BACKUP_DEST="${BACKUP_DEST:-/mnt/nas/backups}"
FASTCOPY_PROFILE="${FASTCOPY_PROFILE:-/etc/fastcopy/backup.yaml}"
ALERT_EMAIL="${ALERT_EMAIL}"

# Timestamp for versioned backups
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
DEST_DIR="${BACKUP_DEST}/${TIMESTAMP}"

# Create profile dynamically
cat > /tmp/backup-${TIMESTAMP}.yaml <<EOF
transfer:
  mode: "turbo"
  streams: 32
  verify: true
  resume: always

paths:
  source: "${BACKUP_SOURCE}"
  destination: "${DEST_DIR}"

logging:
  level: "info"
  file: "/var/log/fastcopy/backup-${TIMESTAMP}.log"

hooks:
  on_complete: "echo 'Backup completed at ${DEST_DIR}' | mail -s 'Backup Success' ${ALERT_EMAIL}"
  on_error: "echo 'Backup FAILED - Check logs' | mail -s 'BACKUP FAILURE' ${ALERT_EMAIL}"
EOF

# Execute backup
fastcopy --profile /tmp/backup-${TIMESTAMP}.yaml --priority high

# Cleanup old backups (keep last 7 days)
find "${BACKUP_DEST}" -type d -mtime +7 -exec rm -rf {} +

Watch Directory for Changes

# watch-sync.yaml
transfer:
  mode: "balanced"
  streams: 16
  verify: true
  resume: on_failure

paths:
  source: "/home/user/Documents"
  destination: "/mnt/cloud-sync/Documents"

schedule:
  type: "watch"                # monitor source for changes
  interval_seconds: 60         # check every 60 seconds
  debounce_ms: 5000           # wait 5s after last change

filters:
  exclude:
    - ".DS_Store"
    - "*.tmp"
    - "~*"

hooks:
  on_complete: "notify-send 'Documents synced'"

Parallel Multi-Source Backup

#!/bin/bash
# multi-source-backup.sh

# Define multiple sources
SOURCES=(
  "/var/www/app1:/backup/app1"
  "/var/www/app2:/backup/app2"
  "/home/user/data:/backup/user-data"
)

# Run transfers in parallel
for source_dest in "${SOURCES[@]}"; do
  IFS=':' read -r source dest <<< "$source_dest"
  
  fastcopy "$source" "$dest" \
    --streams 16 \
    --verify \
    --log-file "/var/log/fastcopy/$(basename $source).log" &
done

# Wait for all transfers to complete
wait

echo "All backups completed"

Configuration Files

Global Configuration

# ~/.fastcopy/config.yaml or /etc/fastcopy/config.yaml
global:
  default_mode: "balanced"
  default_streams: 16
  buffer_size_mb: 128
  temp_directory: "/tmp/fastcopy"
  
network:
  timeout_seconds: 300
  retry_attempts: 3
  retry_backoff_ms: 1000

integrity:
  algorithm: "sha256"          # sha256, crc32, both
  verify_on_write: true
  
performance:
  cpu_affinity: true           # pin threads to cores
  io_priority: "normal"        # low, normal, high, realtime
  
logging:
  default_level: "info"
  console_output: true
  log_directory: "/var/log/fastcopy"

API Integration Examples

OpenAI Integration

# ai-categorized-transfer.yaml
transfer:
  mode: "turbo"
  streams: 32

paths:
  source: "/incoming/media"
  destination: "/organized/media"

ai:
  provider: "openai"
  endpoint: "https://api.openai.com/v1"
  api_key_env: "OPENAI_API_KEY"
  model: "gpt-4-turbo"
  
  categorization:
    enabled: true
    prompt: "Analyze this filename and suggest a category folder: {filename}"
    auto_organize: true

Claude Integration

# claude-documentation.yaml
transfer:
  mode: "balanced"
  streams: 16

paths:
  source: "/project/codebase"
  destination: "/backup/codebase"

ai:
  provider: "claude"
  endpoint: "https://api.anthropic.com"
  api_key_env: "ANTHROPIC_API_KEY"
  model: "claude-3-opus-20240229"
  
  documentation:
    enabled: true
    generate_summary: true
    output_file: "/backup/transfer-report.md"

Troubleshooting

Common Issues

Transfer Hangs or Stalls

# Reduce concurrent streams
fastcopy source/ dest/ --streams 8

# Lower buffer size for memory-constrained systems
fastcopy source/ dest/ --buffer-size-mb 64

# Check system resources
fastcopy source/ dest/ --log-level debug

Permission Errors

# Run with elevated privileges (use cautiously)
sudo fastcopy /protected/source /backup/

# Or adjust ownership/permissions beforehand
sudo chown -R $USER:$USER /destination/

Integrity Check Failures

# Re-run with verification enabled
fastcopy source/ dest/ --verify --retry-on-error

# Use both checksums for maximum reliability
fastcopy source/ dest/ --integrity both --streams 8

# Check source file system for corruption
fsck /dev/source_device

Network Transfer Issues

# Increase timeout for slow networks
fastcopy //remote/share /local/ --timeout 600

# Reduce streams for unstable connections
fastcopy //remote/share /local/ --streams 4 --retry-attempts 5

# Use resume mode for unreliable connections
fastcopy //remote/share /local/ --resume always

Logging and Debugging

# Enable maximum verbosity
fastcopy source/ dest/ --log-level debug --console-output

# Write detailed logs to file
fastcopy source/ dest/ --log-file /tmp/transfer.log --log-level debug

# Monitor transfer statistics
fastcopy source/ dest/ --stats realtime --progress bar

Performance Tuning

# Maximum performance (high CPU/memory usage)
fastcopy large_dataset/ /backup/ \
  --mode turbo \
  --streams 64 \
  --buffer-size-mb 512 \
  --priority realtime \
  --no-verify

# Balanced (recommended for most use cases)
fastcopy data/ /backup/ \
  --mode balanced \
  --streams 16 \
  --verify

# Low resource usage
fastcopy source/ dest/ \
  --mode standard \
  --streams 4 \
  --buffer-size-mb 64 \
  --priority low

Environment Variables

# Configuration
export FASTCOPY_CONFIG=/path/to/config.yaml
export FASTCOPY_PROFILE=/path/to/default-profile.yaml

# Credentials for AI integrations
export OPENAI_API_KEY=sk-...
export ANTHROPIC_API_KEY=sk-ant-...

# Network settings
export FASTCOPY_TIMEOUT=600
export FASTCOPY_RETRY_ATTEMPTS=5

# Performance tuning
export FASTCOPY_STREAMS=32
export FASTCOPY_BUFFER_MB=256

# Logging
export FASTCOPY_LOG_LEVEL=info
export FASTCOPY_LOG_DIR=/var/log/fastcopy

Best Practices

1. Always enable verification for critical data: Use --verify flag
2. Use profiles for repeated operations: Maintain YAML configs in version control
3. Start with fewer streams and scale up: Test with --streams 8 before increasing
4. Monitor system resources during large transfers: Use htop or Task Manager
5. Implement proper error handling in automation: Use hooks for alerting
6. Keep logs for audit trails: Configure persistent logging
7. Test restoration procedures regularly: Verify backups are recoverable
8. Use appropriate mode for workload: Don't always use turbo mode