File Organization

File Organization

Purpose

Bring order to a directory without losing anything. Every file-organization task has one hard requirement that overrides all others: nothing is destroyed, and every action is reversible.

When to Use

• Organizing a directory of accumulated files.
• Finding and removing duplicates.
• Reclaiming disk space.
• Applying a consistent naming convention.
• Sorting downloads, documents, or media.

Capabilities

• Classification by type, content, and date.
• True duplicate detection by content hash.
• Consistent renaming.
• Space analysis.
• Safe, reversible operations.

Inputs

• The directory, and how large it actually is.
• The organizing principle: by type, by date, by project.
• What must not be touched.

Outputs

• A plan, shown before anything is executed.
• The reorganization, performed reversibly.
• A report of what changed.

Workflow

1. Survey first — Count, size, types, and the largest items. Never act on a directory you have not looked at.
2. Propose a plan and show it — What will move where, what will be renamed, what will be deleted. This is shown, and confirmed, before anything happens.
3. Detect duplicates by content, not by name — Hash the files. Two files with the same name are frequently different; two files with different names are frequently identical.
4. Move, do not delete — Duplicates and junk go to a quarantine directory, not to oblivion. Delete only after the user has confirmed, and preferably never.
5. Preserve the metadata — Modification times carry information. Do not destroy them by copying carelessly.
6. Report what happened — With a way to undo it.

Best Practices

• Never delete. Move to a quarantine directory and let the user delete it themselves once they are satisfied. A cleanup that destroys one needed file has failed regardless of how much space it reclaimed.
• Hash before declaring a duplicate. Filename matching produces false positives (two different invoice.pdf) and false negatives (report.pdf and report (1).pdf that are identical).
• Hash cheaply: compare sizes first, then hash only the files whose sizes match. Hashing a 200 GB directory in full is slow and almost entirely wasted.
• Dry-run everything. Show the plan, and require confirmation for anything destructive.
• Preserve the original name somewhere — in a manifest, if not in the filename. Renaming without a record is irreversible in practice.
• Watch for hard links and symlinks. "Deleting a duplicate" that is a hard link frees nothing and may break something.

Examples

Duplicate detection that is correct and fast:

import hashlib
from collections import defaultdict
from pathlib import Path

def find_duplicates(root: Path) -> dict[str, list[Path]]:
    """Two passes. Size first — cheap. Hash only the size collisions."""
    by_size: dict[int, list[Path]] = defaultdict(list)

    for path in root.rglob("*"):
        if path.is_file() and not path.is_symlink():
            by_size[path.stat().st_size].append(path)

    # Only files with an identical size can be identical. Everything else is
    # excluded without reading a byte.
    candidates = [paths for paths in by_size.values() if len(paths) > 1]

    by_hash: dict[str, list[Path]] = defaultdict(list)
    for group in candidates:
        for path in group:
            by_hash[_hash(path)].append(path)

    return {h: paths for h, paths in by_hash.items() if len(paths) > 1}


def _hash(path: Path, chunk: int = 1 << 20) -> str:
    h = hashlib.blake2b(digest_size=16)
    with path.open("rb") as f:
        while data := f.read(chunk):
            h.update(data)
    return h.hexdigest()

A plan shown before anything is done:

Survey of ~/Downloads:
  2,847 files, 41.2 GB

  Duplicates (identical content) : 312 files, 8.4 GB reclaimable
  Installers (.dmg, .pkg, .exe)  : 89 files, 14.1 GB — all older than 6 months
  Screenshots                    : 1,204 files, 2.1 GB
  Documents (pdf, docx)          : 418 files
  Archives (.zip, .tar.gz)       : 203 files, 9.8 GB

PLAN — nothing is deleted. Everything is moved, and a manifest records the
original location of every file.

  1. Duplicates -> ~/Downloads/_quarantine/duplicates/
     The most recently modified copy of each set stays where it is.
     312 files, 8.4 GB.

  2. Installers older than 6 months -> ~/Downloads/_quarantine/installers/
     89 files, 14.1 GB.

  3. Screenshots -> ~/Downloads/Screenshots/YYYY-MM/
     Organized by capture date (from EXIF where available, mtime otherwise).

  4. Documents -> ~/Downloads/Documents/
  5. Archives  -> left in place (they may be needed; too risky to move blind).

  Manifest written to ~/Downloads/_quarantine/manifest.json.
  To undo everything:  python restore.py manifest.json

Proceed? [y/N]

Notes

• The size-then-hash approach is not an optimization detail; on a large directory it is the difference between a task that finishes in seconds and one that reads every byte on the disk.
• The quarantine directory plus a manifest is what makes a cleanup safe. The user gets the space back the moment they empty it, and until then nothing is lost.
• BLAKE2 is faster than SHA-256 and entirely adequate for duplicate detection, where you are not defending against an adversary constructing collisions.