Stage Conversion

Replace Embedded File Paths in Snowpark Code

Scans SMA-converted Snowpark code for embedded file paths and updates them. Handles two scenarios:

1. Already converted paths (@STAGE/protocol/path) - Change stage prefix
2. Unconverted paths (original protocols like s3://, hdfs://, relative paths) - Convert to stage format

Uses script: scripts/embedded_path_replacer.py for AST-based path detection and transformation.

Output Format

Every time you begin a step, sub-step, or significant action, prefix the message with a timestamp in the format [YYYY-MM-DD HH:MM:SS]. Obtain the current time by running date '+%Y-%m-%d %H:%M:%S' in bash.

Example:

[2026-03-24 14:05:32] Starting Step 1: Identify target files...
[2026-03-24 14:05:45] Found 8 files with embedded paths
[2026-03-24 14:05:46] Step 1 complete.

Workflow Checklist

Copy this checklist and track your progress:

Path Replacement Progress:
- [ ] Step 0: Initialize Git
- [ ] Step 1: Identify target files
- [ ] Step 2: Scan for embedded paths
- [ ] Step 3: Present findings and get prefix from user
- [ ] Step 4: Preview changes (dry-run)
- [ ] Step 5: Get user approval
- [ ] Step 6: Apply replacements
- [ ] Step 7: Generate report and summary
- [ ] Step 8: Verify results
- [ ] Step 9: Commit changes to Git

Step 0: Initialize Git

Ensure the workload directory has a git repository on the sma/migration-process branch. This is idempotent — if the orchestrator already initialized git, this is a no-op.

result = sma_api.git_ensure_ready("<workload_path>")

Step 1: Identify Target Files

Goal: Determine which files to scan for embedded paths.

Ask user:

Which files should I scan for embedded file paths?

Options:
1. Specific file(s) - provide path(s)
2. Directory - scan all Python files recursively
3. Current file - if already working on one

Actions:

1. Get file/directory path from user
2. If directory, glob for **/*.py and **/*.ipynb files
3. List files that will be scanned and confirm with user

Completion criteria: User has confirmed the list of files to scan.

Step 2: Scan for Embedded Paths

Goal: Detect all embedded file paths in the target files.

Action: Run the scanner script in scan-only mode:

python3 scripts/embedded_path_replacer.py --scan-only [files...]

Note: If script is not executable, run: chmod +x scripts/embedded_path_replacer.py

What the script detects:

Completion criteria: Script has returned list of detected paths with locations.

Step 3: Present Findings and Get Prefix

Goal: Show user what was found and get the stage prefix for transformation.

Present findings to user:

Found N embedded file paths in M files:

File: data_processor.py
  - s3://company-data/sales.csv (line 45, Snowpark read)
  - hdfs://cluster/input.parquet (line 67, Snowpark read)

File: analysis.ipynb
  - ./config/settings.json (cell[3]:line 12, String variable)

Explain transformation format:

Paths will be transformed to: @{PREFIX}/{protocol}/{path}

Examples:
  s3://bucket/file.csv      -> @PREFIX/s3/bucket/file.csv
  hdfs://cluster/data       -> @PREFIX/hdfs/cluster/data
  /local/path/file.csv      -> @PREFIX/local/path/file.csv
  ./relative/path           -> @PREFIX/relative/path
  ../parent/path            -> @PREFIX/relative/parent/path

Ask user:

What prefix should be used for all file path replacements?
(e.g., MY_STAGE, DATA_STAGE, INPUT_FILES)

STOP: Wait for user to provide the prefix before continuing.

Completion criteria: User has provided a stage prefix.

Step 4: Preview Changes (Dry Run)

Goal: Show exactly what will be changed before making modifications.

Action: Run script in dry-run mode:

python3 scripts/embedded_path_replacer.py --dry-run --prefix {USER_PREFIX} [files...]

Present preview to user:

Planned replacements with prefix "@{USER_PREFIX}":

  s3://company-data/sales.csv
  -> @{USER_PREFIX}/s3/company-data/sales.csv
  Occurrences: 3 (in 2 files)

  hdfs://cluster/input.parquet
  -> @{USER_PREFIX}/hdfs/cluster/input.parquet
  Occurrences: 1

  ./config/settings.json
  -> @{USER_PREFIX}/relative/config/settings.json
  Occurrences: 1

Total: N replacements across M files

Completion criteria: Preview has been shown to user.

Step 5: Get User Approval

Goal: Obtain explicit approval before modifying files.

Ask user:

Apply these changes? (Yes / No / Modify prefix)

STOP: Do NOT proceed without explicit "Yes" from user.

If user says "No": End workflow, no changes made. If user says "Modify prefix": Return to Step 3. If user says "Yes": Continue to Step 6.

Completion criteria: User has explicitly approved the changes.

Step 6: Apply Replacements

Goal: Execute the path transformations on all target files.

Action: Run script to apply changes:

python3 scripts/embedded_path_replacer.py --prefix {USER_PREFIX} [files...]

What the script does:

1. Reads each file (Python or Jupyter notebook)
2. Replaces paths while preserving quote style (' or ")
3. Injects WARNING comments for paths marked needs_revision
4. Writes modified content back to file
5. Reports results per file

Monitor output:

Applying replacements...

Results:
  data_processor.py: 3 replacements
  analysis.ipynb: 2 replacements
  etl_pipeline.py: 5 replacements

Total: 10 replacements applied
Warnings injected: 2 (dynamic paths)

Completion criteria: Script has completed without errors.

Step 7: Generate Report and Summary

Goal: Create audit trail and provide summary to user.

Reports generated automatically:

CSV columns:

• Original Path
• Transformed Path
• Status (replaced, needs_revision, skipped_*)
• Reason
• Detection Method
• File
• Location
• Line Content

Present summary to user:

Replacement complete!

Results:
  Replaced: N paths in M files
  Needs revision: X paths (dynamic/f-string)
  Skipped: Y paths

Files modified:
  - data_processor.py (3 replacements)
  - analysis.ipynb (2 replacements)

Reports generated:
  - sma_path_replacement_report_20260305_103000.csv
  - sma_path_replacement_report_20260305_103000.json

Completion criteria: Reports generated and summary presented.

Step 8: Verify Results

Goal: Ensure transformations were applied correctly and guide next steps.

Verification actions:

1. Check that replaced count matches expected count from preview
2. Review any paths marked needs_revision in the report
3. Identify any paths that were skipped and explain why

Present next steps to user:

Next steps:
1. Review paths marked "needs_revision" - these contain dynamic elements
2. Check the CSV report for full details
3. Test the modified code to ensure it works correctly

If there are needs_revision paths:

Paths requiring manual review:
  - @PREFIX/s3/{bucket}/data.csv (contains f-string variable)
  - @PREFIX/hdfs/{cluster}/table (multiple variables)

These paths were transformed but contain dynamic elements.
Please verify the transformations are correct for your use case.

Completion criteria: User has been informed of results and next steps.

Step 9: Commit Changes to Git

After all replacements are applied and verified, commit the changes:

result = sma_api.git_commit("<workload_path>", """DVP Stage Conversion: Replaced N paths in M files

Prefix: @USER_PREFIX
Replaced: N paths
Needs revision: X (dynamic/f-string paths)
Reports: sma_path_replacement_report_YYYYMMDD_HHMMSS.csv/.json""")

Verify branches:

result = sma_api.git_verify_branches("<workload_path>")

Quick Reference

Supported Path Patterns

CLI Commands

# Scan only (no changes)
python3 scripts/embedded_path_replacer.py --scan-only [files...]

# Dry run (preview changes)
python3 scripts/embedded_path_replacer.py --dry-run --prefix PREFIX [files...]

# Apply changes
python3 scripts/embedded_path_replacer.py --prefix PREFIX [files...]

# Apply without warnings
python3 scripts/embedded_path_replacer.py --prefix PREFIX --no-warnings [files...]

# Skip git check (automated workflows)
python3 scripts/embedded_path_replacer.py --prefix PREFIX --skip-git-check [files...]

Status Values

Important Notes

• Prefix format: Do NOT include @ in your prefix - it's added automatically
• Quote preservation: The script preserves original quote style (' or ")
• Jupyter notebooks: Only Python code cells are processed
• Dynamic paths: F-strings like s3://{bucket}/path are transformed but flagged for review
• Git recommended: Always use version control before bulk modifications
• Idempotent: Running the script twice with the same prefix produces the same result

Final Summary

MANDATORY: After completing all steps (whether running standalone or invoked from the orchestrator), ALWAYS present this summary table:

Stage Conversion Complete

┌──────────────────────┬──────────┬──────────────────────────────────────────────────┐
│ Step                 │ Status   │ Details                                          │
├──────────────────────┼──────────┼──────────────────────────────────────────────────┤
│ Stage Conversion     │ Done     │ Replaced paths in N files                        │
└──────────────────────┴──────────┴──────────────────────────────────────────────────┘

Output location: <output>/

Git branches:
• main — original code (unmodified)
• sma/migration-process — stage conversion changes applied

Rules:

• Replace N with actual count of files where paths were replaced
• Status is Done, Skipped, or Failed
• If no embedded paths were found, show Skipped with reason (e.g., "No embedded paths found")
• If conversion failed, show Failed with brief error
• The git branches section uses sma_api.git_verify_branches() to confirm both branches exist