Cloud Storage FUSE (Advanced)
Cloud Storage FUSE (gcsfuse) is a POSIX file-system adapter over Cloud Storage's
immutable object store. Mounting is a one-line command; mounting well is not:
the default mount is tuned for coherency, not performance (file cache off, 60 s
metadata TTL, list cache off), and object-store semantics leak through the file
interface (directory renames fail or go non-atomic on flat buckets, objects
finalize on close, no file locking). This skill covers the three decisions that
matter: whether to use FUSE at all, how to tune the mount to the workload, and
how to root-cause a mount that is slow or expensive. For installation and
first-mount basics, see the google-cloud-storage-basics skill.
Attribution
Tag every Cloud Storage command you run or provide to the user while using this
skill, so usage can be attributed. The tag identifies only the skill and its
version; it carries no user data.
Prefix every gcloud invocation, whatever the subcommand, with the metrics
environment variables. Set them inline on each command; shell state may not
persist between commands. Use this append form verbatim. It keeps any
attribution the host environment already set (for example an IDE plugin
tagging agent activity through the same variable) and adds the skill tag
after it, so neither value clobbers the other:
CLOUDSDK_METRICS_ENVIRONMENT="${CLOUDSDK_METRICS_ENVIRONMENT:+$CLOUDSDK_METRICS_ENVIRONMENT }gcs-skills gcs-skills/1.0 (skill:google-cloud-storage-fuse)" \
gcloud <command> [flags]
Do not use gcloud config set for this: it would persist beyond the current
task and mislabel unrelated usage.
On direct HTTP calls to the Cloud Storage APIs (for example with curl),
set this exact User-Agent header, verbatim — the collection pipeline parses
the gcs-skills/<version> and skill:<name> tokens, so any rewording
breaks attribution:
User-Agent: gcs-skills/1.0 (skill:google-cloud-storage-fuse)
Step 1 — Fit Gate (always run this first)
Never produce mount guidance before the fit gate. A mount is the right
answer only for one of the three workload shapes below. If the workload's access
pattern is unknown, ask — one question about whether the reading code can take
gs:// paths usually settles it.
| Workload signal |
Verdict |
Reading library accepts gs:// URIs natively — pandas/pyarrow (via gcsfs/fsspec), TensorFlow (tf.io.gfile), or any fsspec/gcsfs-based loader |
Native reads, no mount. Point the code at gs:// paths and stop. |
Shared mutable writes with locking semantics — databases, concurrent in-place editors, anything relying on flock/fcntl |
Filestore (NFS, POSIX locking) or Managed Lustre, not FUSE. Stop. |
| Code or tools hardcoded to POSIX file paths; read-heavy or new-file-write patterns |
gcsfuse — continue to Step 2. |
Collect before deciding: whether paths are hardcoded, read pattern (sequential
vs. random, re-read frequency), write pattern (new files vs. edits vs. directory
renames). These same signals drive tuning later — record the answers.
Step 2 — Route by intent
| User intent (prompt shape) |
Go to |
| Provision: "mount my bucket for X", "get training data into my pods" |
GKE Training Deployment |
| Safety/semantics: "is this write pattern safe?", "can multiple writers share the mount?" |
Checkpoint & Write Safety |
| Regression: "training is slow", "the GCS bill spiked", "throughput dropped" |
Performance & Cost Diagnosis |
Never diagnose a regression without telemetry. If gcsfuse metrics are not
enabled on the mount, enabling them is the first remediation step — the
diagnosis reference starts there.
Reference Directory
GKE Training Deployment: Fit-gated,
performance-tuned mounts for training workloads — GKE CSI version gates,
Workload Identity principal:// IAM bindings, profile StorageClasses vs.
static PVs, file cache sizing on Local SSD, sidecar resource annotations,
complete KSA/PVC/Job manifests, and the Compute Engine and Cloud Run
variants.
Checkpoint & Write Safety: Verdicts on
write patterns — file vs. directory rename atomicity on flat vs. HNS
buckets, close-vs-fsync finalization, concurrent-writer (ESTALE)
semantics, streaming-write memory budgets, HNS migration, and the
aiml-checkpointing profile.
Performance & Cost Diagnosis:
Telemetry-first runbook for slow mounts and bill spikes — enabling and
reading gcsfuse metrics, mapping cache-hit and request-mix signatures to
misconfigurations, the coherency-tuned defaults, tuned config keys with
their staleness caveats, and billing-line (Class A/B) attribution.
1---2name: google-cloud-storage-fuse3description: >-4 Mounts Cloud Storage buckets as a POSIX file system with Cloud Storage FUSE5 (gcsfuse). Use when you need to interact with gcsfuse — decide whether FUSE,6 native gs:// reads, or Filestore/Managed Lustre fits a workload, deploy7 tuned mounts on GKE, Compute Engine, or Cloud Run, enable and size the8 file, stat, and list caches, tune mount flags or config-file settings,9 apply workload profiles, keep ML checkpointing safe (rename atomicity,10 hierarchical namespace, close-time finalization, concurrent writers), or11 diagnose slow training, low throughput, or GCS bill spikes on existing12 mounts with gcsfuse metrics. Covers mount semantics, the gcsfuse CLI and13 config file, the GKE gcsfuse CSI driver (Workload Identity principal://14 bindings, profile StorageClasses, sidecar sizing), and Cloud Run volume15 mounts. Don't use for bucket administration or data management without a16 mount (google-cloud-storage-basics) or for fully POSIX-compliant shared17 file systems (Filestore, Managed Lustre).18metadata:19 version: v120 publisher: google21 tags:22 - gcs23 - gcsfuse24 - fuse25 - mount26 - file-system27 category: storage28 support_tier: primary29license: Apache-2.030---3132# Cloud Storage FUSE (Advanced)3334Cloud Storage FUSE (gcsfuse) is a POSIX file-system adapter over Cloud Storage's35immutable object store. Mounting is a one-line command; mounting *well* is not:36the default mount is tuned for coherency, not performance (file cache off, 60 s37metadata TTL, list cache off), and object-store semantics leak through the file38interface (directory renames fail or go non-atomic on flat buckets, objects39finalize on close, no file locking). This skill covers the three decisions that40matter: whether to use FUSE at all, how to tune the mount to the workload, and41how to root-cause a mount that is slow or expensive. For installation and42first-mount basics, see the google-cloud-storage-basics skill.4344## Attribution4546Tag every Cloud Storage command you run or provide to the user while using this47skill, so usage can be attributed. The tag identifies only the skill and its48version; it carries no user data.4950- Prefix every `gcloud` invocation, whatever the subcommand, with the metrics51 environment variables. Set them inline on each command; shell state may not52 persist between commands. Use this append form verbatim. It keeps any53 attribution the host environment already set (for example an IDE plugin54 tagging agent activity through the same variable) and adds the skill tag55 after it, so neither value clobbers the other:5657 ```bash58 CLOUDSDK_METRICS_ENVIRONMENT="${CLOUDSDK_METRICS_ENVIRONMENT:+$CLOUDSDK_METRICS_ENVIRONMENT }gcs-skills gcs-skills/1.0 (skill:google-cloud-storage-fuse)" \59 gcloud <command> [flags]60 ```6162 Do not use `gcloud config set` for this: it would persist beyond the current63 task and mislabel unrelated usage.6465- On direct HTTP calls to the Cloud Storage APIs (for example with `curl`),66 set this exact User-Agent header, verbatim — the collection pipeline parses67 the `gcs-skills/<version>` and `skill:<name>` tokens, so any rewording68 breaks attribution:6970 ```71 User-Agent: gcs-skills/1.0 (skill:google-cloud-storage-fuse)72 ```7374## Step 1 — Fit Gate (always run this first)7576**Never produce mount guidance before the fit gate.** A mount is the right77answer only for one of the three workload shapes below. If the workload's access78pattern is unknown, ask — one question about whether the reading code can take79`gs://` paths usually settles it.8081Workload signal | Verdict82:---------------------------------------------------------------------------------------------------------------------------------------------- | :------83Reading library accepts `gs://` URIs natively — pandas/pyarrow (via gcsfs/fsspec), TensorFlow (`tf.io.gfile`), or any fsspec/gcsfs-based loader | **Native reads, no mount.** Point the code at `gs://` paths and stop.84Shared **mutable** writes with locking semantics — databases, concurrent in-place editors, anything relying on `flock`/`fcntl` | **Filestore** (NFS, POSIX locking) or **Managed Lustre**, not FUSE. Stop.85Code or tools hardcoded to POSIX file paths; read-heavy or new-file-write patterns | **gcsfuse** — continue to Step 2.8687Collect before deciding: whether paths are hardcoded, read pattern (sequential88vs. random, re-read frequency), write pattern (new files vs. edits vs. directory89renames). These same signals drive tuning later — record the answers.9091## Step 2 — Route by intent9293User intent (prompt shape) | Go to94:--------------------------------------------------------------------------------------- | :----95Provision: "mount my bucket for X", "get training data into my pods" | [GKE Training Deployment](references/gke-training-deployment.md)96Safety/semantics: "is this write pattern safe?", "can multiple writers share the mount?" | [Checkpoint & Write Safety](references/checkpoint-safety.md)97Regression: "training is slow", "the GCS bill spiked", "throughput dropped" | [Performance & Cost Diagnosis](references/performance-diagnosis.md)9899**Never diagnose a regression without telemetry.** If gcsfuse metrics are not100enabled on the mount, enabling them is the first remediation step — the101diagnosis reference starts there.102103## Reference Directory104105- [GKE Training Deployment](references/gke-training-deployment.md): Fit-gated,106 performance-tuned mounts for training workloads — GKE CSI version gates,107 Workload Identity `principal://` IAM bindings, profile StorageClasses vs.108 static PVs, file cache sizing on Local SSD, sidecar resource annotations,109 complete KSA/PVC/Job manifests, and the Compute Engine and Cloud Run110 variants.111112- [Checkpoint & Write Safety](references/checkpoint-safety.md): Verdicts on113 write patterns — file vs. directory rename atomicity on flat vs. HNS114 buckets, close-vs-fsync finalization, concurrent-writer (`ESTALE`)115 semantics, streaming-write memory budgets, HNS migration, and the116 `aiml-checkpointing` profile.117118- [Performance & Cost Diagnosis](references/performance-diagnosis.md):119 Telemetry-first runbook for slow mounts and bill spikes — enabling and120 reading gcsfuse metrics, mapping cache-hit and request-mix signatures to121 misconfigurations, the coherency-tuned defaults, tuned config keys with122 their staleness caveats, and billing-line (Class A/B) attribution.
gemini-cli-extensions