Build Flydsl

Build and Install FlyDSL

Build FlyDSL from source on a remote host or Docker container. FlyDSL requires a custom LLVM/MLIR build with Python bindings, followed by the FlyDSL C++ dialect and Python package.

Arguments

Checkout path

The steps below use $FLYDSL_ROOT for the checkout. Set it before starting; the container images this skill targets conventionally clone to /FlyDSL, but a developer checkout is wherever it was cloned:

export FLYDSL_ROOT="${FLYDSL_ROOT:-$PWD}"   # or /FlyDSL inside the prebuilt image

Prerequisites

• ROCm 6.x or 7.x (for GPU execution)
• cmake >= 3.20, C++17 compiler, ninja (recommended)
• Python 3.10+ with pip
• Git (to clone LLVM)
• Disk space: ~50GB for LLVM build + FlyDSL
• CPU cores: more is better (use -j$(nproc) or -j128)

Build Steps

Step 1: Build LLVM/MLIR (~30 min with -j128)

This clones ROCm/llvm-project, checks out the commit specified in thirdparty/llvm-hash.txt, and builds MLIR with Python bindings.

cd "$FLYDSL_ROOT"
bash scripts/build_llvm.sh -j128

What it does:

1. Clones https://github.com/ROCm/llvm-project.git to ../llvm-project/
2. Checks out the pinned commit from thirdparty/llvm-hash.txt
3. CMake configure + build + install to ../llvm-project/mlir_install/
4. Creates tarball ../llvm-project/mlir_install.tgz

If you already have an MLIR build, skip this step and set:

export MLIR_PATH=/path/to/llvm-project/mlir_install

Step 2: Build FlyDSL (~5 min)

cd "$FLYDSL_ROOT"
bash scripts/build.sh -j128

What it does:

1. Auto-detects MLIR_PATH from common locations (or uses env var)
2. CMake configure + build the Fly dialect (C++) and Python bindings
3. Output: build-fly/python_packages/flydsl/ with embedded MLIR bindings

Step 3: Install (editable mode)

cd "$FLYDSL_ROOT"
pip install -e .

Or without installing (just set paths):

export PYTHONPATH="$FLYDSL_ROOT/build-fly/python_packages:$FLYDSL_ROOT:$PYTHONPATH"
export LD_LIBRARY_PATH="$FLYDSL_ROOT/build-fly/python_packages/flydsl/_mlir/_mlir_libs:$LD_LIBRARY_PATH"

Step 4: Verify

python3 -c "import flydsl; print('FlyDSL OK')"
bash scripts/run_tests.sh  # GEMM correctness tests (~15s)

Remote/Docker Execution

For building inside a Docker container on a remote host:

# SSH command pattern
ssh -o LogLevel=ERROR <HOST> 'docker exec <CONTAINER> bash -c "cd "$FLYDSL_ROOT" && CMD"'

# Full build sequence (run each step, wait for completion)
ssh ... 'docker exec -d <CONTAINER> bash -c "cd "$FLYDSL_ROOT" && bash scripts/build_llvm.sh -j128 > /tmp/build_llvm.log 2>&1"'
# Monitor: ssh ... 'docker exec <CONTAINER> tail -5 /tmp/build_llvm.log'
# Wait for "LLVM_BUILD_DONE" or "Creating tarball..." followed by completion

ssh ... 'docker exec <CONTAINER> bash -c "cd "$FLYDSL_ROOT" && bash scripts/build.sh -j128"'
ssh ... 'docker exec <CONTAINER> bash -c "cd "$FLYDSL_ROOT" && pip install -e ."'
ssh ... 'docker exec <CONTAINER> bash -c "python3 -c \"import flydsl; print(\\\"FlyDSL OK\\\")\""'

Build Artifacts (Do NOT Commit)

The following directories are build artifacts generated by CMake and should NOT be committed to git:

• python/flydsl/_mlir/ — MLIR Python bindings (compiled .so files, platform-specific); absent until you build
• build/, build-fly/ — CMake build directories

These are already in .gitignore. After a fresh clone, run the build steps above to regenerate.

Rebuild After Code Changes

# C++ changes (dialect, MLIR passes):
bash scripts/build.sh -j128

# Python-only changes:
# No rebuild needed — editable install picks up changes automatically.

# Clear kernel cache if stale results:
rm -rf ~/.flydsl/cache
# Or disable disk cache (in-memory cache remains active):
export FLYDSL_RUNTIME_ENABLE_CACHE=0

Troubleshooting

• std::gcd not found or redeclaration errors: Wrong LLVM picked up. unset MLIR_PATH and let build.sh auto-detect.
• No module named flydsl: Run pip install -e . or set PYTHONPATH.
• MLIR .so load errors: Set LD_LIBRARY_PATH to include build-fly/python_packages/flydsl/_mlir/_mlir_libs/.
• Build OOM: Reduce parallelism (e.g., -j64 instead of -j128).
• Docker exec -d for long builds: Use background mode and monitor log file. LLVM build takes ~30min with 128 cores.

Verified Environments