File Management

Source: https://github.com/aipoch/medical-research-skills

File Management

When to Use

• You need to reorganize a directory (move/copy/rename) based on include/exclude patterns and a target folder structure.
• You want a repeatable backup workflow (copy or archive) with optional retention rules and verification manifests.
• You need to compress a folder or a set of files to reduce size or package deliverables.
• You must split very large files into fixed-size parts for transfer/storage limits (binary-safe).
• You need to merge previously split parts back into the original file reliably (binary-safe).

Key Features

• Safety-first planning: generate a preview/manifest of intended actions before writing changes.
• Rule-driven scope control: root path + include/exclude patterns + thresholds + target structure (see references/rule-schema.md).
• Conflict handling: default to skip/rename on collisions; overwrite only with explicit confirmation.
• Non-destructive defaults: avoid deletion unless explicitly requested.
• Verification support: compare counts/sizes and optionally generate hashes for critical backups.
• Binary-safe split/merge: split by fixed size and merge parts without corrupting binary data (scripts/).

Dependencies

• PowerShell 5.1+ (Windows PowerShell) or PowerShell 7+
  • Cmdlets used: Get-ChildItem, Copy-Item, Move-Item, Compress-Archive, Get-FileHash
• Python 3.8+ (for split/merge scripts)
  • scripts/split_file.py
  • scripts/merge_parts.py

Example Usage

1) Organize files with a preview plan (PowerShell)

# Inputs (adjust to your case)
$Root = "D:\Inbox"
$Target = "D:\Organized"
$Include = @("*.pdf","*.docx")
$ExcludeDirs = @("node_modules",".git")
$WhatIf = $true  # set to $false to execute

# Build a manifest (preview plan)
$files = Get-ChildItem -Path $Root -Recurse -File |
  Where-Object {
    ($Include | ForEach-Object { $_ }) -contains $_.Extension -eq $false
  }

# Practical include filter (simple example)
$files = Get-ChildItem -Path $Root -Recurse -File -Include $Include

# Exclude directories by path segment
$files = $files | Where-Object {
  $p = $_.FullName
  -not ($ExcludeDirs | Where-Object { $p -match [regex]::Escape("\$_\") })
}

$plan = foreach ($f in $files) {
  # Example target structure: by extension
  $ext = $f.Extension.TrimStart(".").ToLower()
  $destDir = Join-Path $Target $ext
  $destPath = Join-Path $destDir $f.Name

  [pscustomobject]@{
    Source = $f.FullName
    Destination = $destPath
    Action = "Copy"  # start with Copy; switch to Move after verification
  }
}

# Preview
$plan | Format-Table -AutoSize

# Execute (safe defaults: create dirs; do not overwrite unless you decide to)
foreach ($item in $plan) {
  $destDir = Split-Path $item.Destination -Parent
  if (-not (Test-Path $destDir)) { New-Item -ItemType Directory -Path $destDir | Out-Null }

  if (Test-Path $item.Destination) {
    # Default conflict strategy: rename
    $base = [IO.Path]::GetFileNameWithoutExtension($item.Destination)
    $ext  = [IO.Path]::GetExtension($item.Destination)
    $dir  = Split-Path $item.Destination -Parent
    $item.Destination = Join-Path $dir ("{0}_{1}{2}" -f $base, (Get-Date -Format "yyyyMMddHHmmss"), $ext)
  }

  if ($item.Action -eq "Copy") {
    Copy-Item -LiteralPath $item.Source -Destination $item.Destination -WhatIf:$WhatIf
  } else {
    Move-Item -LiteralPath $item.Source -Destination $item.Destination -WhatIf:$WhatIf
  }
}

2) Backup + hash manifest (PowerShell)

$Source = "D:\Project"
$BackupRoot = "E:\Backups"
$Stamp = Get-Date -Format "yyyyMMdd_HHmmss"
$BackupDir = Join-Path $BackupRoot ("Project_{0}" -f $Stamp)

# Copy backup
Copy-Item -Path $Source -Destination $BackupDir -Recurse

# Create a hash manifest for verification
$manifestPath = Join-Path $BackupDir "hash-manifest.sha256.txt"
Get-ChildItem -Path $BackupDir -Recurse -File |
  ForEach-Object {
    $h = Get-FileHash -Algorithm SHA256 -LiteralPath $_.FullName
    "{0}  {1}" -f $h.Hash, ($_.FullName.Substring($BackupDir.Length + 1))
  } | Set-Content -Encoding UTF8 $manifestPath

3) Compress a folder (PowerShell)

$SourceDir = "D:\Project"
$ZipPath = "E:\Archives\Project.zip"

Compress-Archive -Path (Join-Path $SourceDir "*") -DestinationPath $ZipPath -Force

4) Split and merge a large file (Python)

# Split into 100MB parts (binary-safe)
python scripts/split_file.py --input "E:\ISO\big.iso" --part-size 100MB --output-dir "E:\ISO\parts"

# Merge parts back (binary-safe)
python scripts/merge_parts.py --input-dir "E:\ISO\parts" --output "E:\ISO\big.iso"

Note: CLI flags may vary depending on the scripts’ implementation. If the scripts do not expose flags, adapt the commands to match their actual parameters.

Implementation Details

• Rule schema and planning

  • Prefer structured rules to define:
    • rootPath
    • include/exclude patterns (glob-like)
    • optional size thresholds (e.g., split if > N MB)
    • target structure mapping (e.g., by extension/date/project)
    • conflict strategy (skip, rename, overwrite)
  • Use references/rule-schema.md as the canonical format for describing these rules.

• Preview-first execution

  • Generate a manifest/plan listing (source, destination, action) before any write operation.
  • Use PowerShell -WhatIf (or an explicit “dry run” mode) to validate the plan.

• Conflict handling

  • Default behavior should be non-destructive:
    • If destination exists: rename (append timestamp or counter) or skip
    • Only overwrite after explicit user confirmation

• Verification

  • Basic checks: file counts and total sizes before/after.
  • Strong checks for critical backups: SHA256 via Get-FileHash, stored as a manifest for later validation.

• Binary-safe splitting/merging

  • Splitting should read/write raw bytes in fixed-size chunks (no text encoding).
  • Merging should concatenate parts in deterministic order (e.g., numeric suffix ordering) to reproduce the original byte stream exactly.