vertex-ai-protein-interleave <<<<<<< HEAD

Vertex AI × Protein-Scale Biology Interleave

Bridge layer connecting Google Cloud's orchestration capabilities (Vertex AI Pipelines, Endpoints, BigQuery) to the ASI protein skill cluster.

description: > Bridge connecting Vertex AI / Google Cloud to protein-scale biology skills. Wires AlphaFold, ESM, DiffDock, DeepChem, and TorchDrug into Vertex AI Pipelines, Endpoints, and BigQuery genomics for protein design-predict-validate loops. Use when orchestrating protein engineering on GCP, deploying ESM as a managed endpoint, querying gnomAD via BigQuery, or running batch docking.

Vertex AI x Protein-Scale Biology Interleave

Bridge connecting Google Cloud's orchestration (Vertex AI Pipelines, Endpoints, BigQuery) to the ASI protein skill cluster.

origin/main

ASI Protein Skill Cluster

<<<<<<< HEAD
Protein Stack (existing in asi)
  ├── alphafold-database (-1)  ← structure retrieval, pLDDT/PAE, 200M+ structures
  ├── esm (-1)                 ← ESM3/ESM C: sequence generation, inverse folding
  ├── diffdock (-1)            ← structure-based docking, pose prediction
  ├── deepchem (0)             ← ADMET prediction, GNNs, MoleculeNet, 30+ datasets
  ├── torchdrug (0)            ← GNNs, retrosynthesis, KG reasoning, 40+ datasets
  ├── adaptyv (+1)             ← wet-lab validation: binding, expression, stability
  ├── uniprot-database (0)     ← search, sequence retrieval, ID mapping
  └── gget (+1)                ← rapid bioinformatics: AlphaFold, ARCHS4, Enrichr

Vertex AI adds: orchestration, serverless inference, genomic warehouse, cost optimization.

GF(3) Tripartite Tag

alphafold-database(-1) ⊗ vertex-ai-protein-interleave(0) ⊗ adaptyv(+1) = 0

Structure (-1) × Bridge (0) × Validation (+1) = balanced protein design loop.

Integration Points

1. Design → Predict → Validate Loop (Vertex AI Pipelines)

The core missing link: orchestrate the full protein engineering iteration via KFP.

alphafold-database (-1) <- structure retrieval, pLDDT/PAE, 200M+ structures esm (-1) <- ESM3/ESM C: sequence generation, inverse folding diffdock (-1) <- structure-based docking, pose prediction deepchem (0) <- ADMET prediction, GNNs, MoleculeNet, 30+ datasets torchdrug (0) <- GNNs, retrosynthesis, KG reasoning, 40+ datasets adaptyv (+1) <- wet-lab validation: binding, expression, stability uniprot-database (0) <- search, sequence retrieval, ID mapping gget (+1) <- rapid bioinformatics: AlphaFold, ARCHS4, Enrichr

Vertex AI adds: orchestration, serverless inference, genomic warehouse, cost optimization.

## 1. Design -> Predict -> Validate Loop (Vertex AI Pipelines)
>>>>>>> origin/main

```python
from kfp import dsl

@dsl.pipeline(name="protein-design-loop")
def protein_pipeline(target_sequence: str, iterations: int = 3):
<<<<<<< HEAD
    # Step 1: AlphaFold structure prediction
=======
>>>>>>> origin/main
    fold = dsl.ContainerOp(
        name="alphafold-predict",
        image="gcr.io/PROJECT/alphafold:latest",
        command=["python", "predict.py"],
        arguments=["--sequence", target_sequence, "--output", "/tmp/structure.pdb"]
    )
<<<<<<< HEAD
    # Step 2: DiffDock binding site prediction
=======
>>>>>>> origin/main
    dock = dsl.ContainerOp(
        name="diffdock",
        image="gcr.io/PROJECT/diffdock:latest",
        command=["python", "inference.py"],
        arguments=["--protein", fold.outputs["structure"], "--ligand", "/data/ligand.sdf"]
    ).after(fold)
<<<<<<< HEAD
    # Step 3: ESM inverse folding → generate sequence variants
=======
>>>>>>> origin/main
    variants = dsl.ContainerOp(
        name="esm-inverse-fold",
        image="gcr.io/PROJECT/esm:latest",
        command=["python", "inverse_fold.py"],
        arguments=["--structure", fold.outputs["structure"], "--num_seqs", "100"]
    ).after(fold)
<<<<<<< HEAD
    # Step 4: DeepChem ADMET filtering
=======
>>>>>>> origin/main
    filtered = dsl.ContainerOp(
        name="deepchem-admet",
        image="gcr.io/PROJECT/deepchem:latest",
        command=["python", "admet_screen.py"],
        arguments=["--sequences", variants.outputs["seqs"]]
    ).after(variants)
<<<<<<< HEAD
    # Step 5: Adaptyv wet-lab order (top candidates)
=======
>>>>>>> origin/main
    dsl.ContainerOp(
        name="adaptyv-order",
        image="gcr.io/PROJECT/adaptyv-client:latest",
        command=["python", "order.py"],
        arguments=["--candidates", filtered.outputs["top_k"], "--assay", "binding"]
    ).after(filtered)

<<<<<<< HEAD Deploy: vertex ai pipelines run --pipeline-spec protein-design-loop.json

2. ESM Serverless Inference via Vertex AI Endpoints

Deploy ESM3 as a managed endpoint for on-demand embedding and sequence generation:

# Build and push ESM container
docker build -t gcr.io/$PROJECT/esm-server:latest -f esm.Dockerfile .
docker push gcr.io/$PROJECT/esm-server:latest

# Upload model artifact
=======
## 2. ESM Serverless Inference via Vertex AI Endpoints

```bash
docker build -t gcr.io/$PROJECT/esm-server:latest -f esm.Dockerfile .
docker push gcr.io/$PROJECT/esm-server:latest

>>>>>>> origin/main
gcloud ai models upload \
  --region=us-central1 \
  --display-name=esm3-protein-lm \
  --container-image-uri=gcr.io/$PROJECT/esm-server:latest \
  --container-predict-route=/predict \
  --container-health-route=/health

<<<<<<< HEAD
# Deploy to endpoint
=======
>>>>>>> origin/main
gcloud ai endpoints create --region=us-central1 --display-name=esm-endpoint
gcloud ai endpoints deploy-model ESM_ENDPOINT_ID \
  --region=us-central1 \
  --model=ESM_MODEL_ID \
  --machine-type=n1-standard-4 \
  --accelerator=count=1,type=nvidia-tesla-t4 \
<<<<<<< HEAD
  --min-replica-count=0 \  # scale to zero when idle
  --max-replica-count=4

# Call endpoint
ACCESS_TOKEN=$(gcloud auth print-access-token)
curl -s "https://us-central1-aiplatform.googleapis.com/v1/projects/${PROJECT}/locations/us-central1/endpoints/${ESM_ENDPOINT_ID}:predict" \
  -H "Authorization: Bearer $ACCESS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"instances": [{"sequence": "MKTAYIAKQRQISFVKSHFSRQLEERLGLIEVQAPILSRVGDGTQDNLSGAEKAVQVKVKALPDAQFEVVHSLAKWKRQTLGQHDFSAGEGLYTHMKALRPDEDRLSPLHSVYVDQWDWERVMGDGERQFSTLKSTVEAIWAGIKATEAAVSEEFGLAPFLPDQIHFVHSQELLSRYPDLDAKGRERAIAKDLGAVFLVGIGGKLSDGHRHDVRAPDYDDWSTPSELGHAGLNGDILVWNPVLEDAFELSSMGIRVDADTLKHQLALTGDEDRLELEWHQALLRGEMPQTIGGGIGQSRLTMLLLQLPHIGQVQAGVWPAAVRESVPSLL"}]}'

3. BigQuery Genomics → Protein ML Pipeline

Wire gnomAD variant data through BigQuery to Vertex AI custom training:

-- Extract high-impact missense variants for protein modeling
-- (gnomAD v3 in BigQuery: bigquery-public-data.gnomAD)
SELECT
  reference_name,
  start_position,
  reference_bases,
  alternate_bases,
  vep.consequence_terms,
  vep.protein_id,
  vep.amino_acids,
=======
  --min-replica-count=0 \
  --max-replica-count=4

3. BigQuery Genomics -> Protein ML Pipeline

-- gnomAD v3 in BigQuery: high-impact missense variants
SELECT
  reference_name, start_position, reference_bases, alternate_bases,
  vep.consequence_terms, vep.protein_id, vep.amino_acids,
>>>>>>> origin/main
  af.AF as allele_frequency
FROM `bigquery-public-data.gnomAD.v3_1_2_genomes`
CROSS JOIN UNNEST(vep) AS vep
CROSS JOIN UNNEST(allele_freq) AS af
WHERE
  'missense_variant' IN UNNEST(vep.consequence_terms)
<<<<<<< HEAD
  AND af.AF > 0.001  -- common variants
=======
  AND af.AF > 0.001
>>>>>>> origin/main
  AND vep.protein_id = @target_protein
ORDER BY af.AF DESC
LIMIT 10000;

<<<<<<< HEAD
# Train Vertex AI custom model on variant-phenotype associations
from google.cloud import aiplatform

aiplatform.init(project=PROJECT, location="us-central1")

=======
from google.cloud import aiplatform

aiplatform.init(project=PROJECT, location="us-central1")
>>>>>>> origin/main
job = aiplatform.CustomTrainingJob(
    display_name="gnomad-variant-phenotype",
    script_path="train_variant_model.py",
    container_uri="us-docker.pkg.dev/vertex-ai/training/pytorch-gpu.1-13:latest",
    requirements=["scikit-learn", "pandas", "biopython"],
)
<<<<<<< HEAD

=======
>>>>>>> origin/main
model = job.run(
    dataset=aiplatform.TabularDataset(DATASET_ID),
    model_display_name="variant-phenotype-predictor",
    machine_type="n1-standard-8",
    accelerator_type="NVIDIA_TESLA_T4",
    accelerator_count=1,
)

<<<<<<< HEAD

4. AlphaFold Batch Inference Pipeline

For large-scale structural characterization using Vertex AI's pre-built AlphaFold integration:

# Use Vertex AI AlphaFold Pipeline (pre-built)
=======
## 4. AlphaFold Batch Inference Pipeline

```bash
>>>>>>> origin/main
gcloud ai pipelines run \
  --pipeline-job-spec-uri=gs://vertex-pipeline-components-public/alphafold/alphafold_pipeline.yaml \
  --parameter-values='
    project='"$PROJECT"',
    region=us-central1,
    input_fasta_gs_path=gs://my-bucket/sequences.fasta,
    output_gs_path=gs://my-bucket/alphafold-output/,
    use_gpu=true,
    model_preset=multimer
  '

<<<<<<< HEAD Process results via alphafold-database skill patterns:

# Load outputs into DuckDB for analysis
=======
```python
>>>>>>> origin/main
import duckdb
conn = duckdb.connect("asi.db")
conn.execute("""
  CREATE TABLE alphafold_batch AS
  SELECT * FROM read_json_auto('gs://my-bucket/alphafold-output/**/*.json')
""")
<<<<<<< HEAD
# Filter by pLDDT confidence
conn.execute("SELECT * FROM alphafold_batch WHERE mean_plddt > 90 ORDER BY mean_plddt DESC")

5. Vertex AI Matching Engine for Protein Similarity Search

Index AlphaFold structure embeddings for semantic protein search:

from google.cloud import aiplatform
import numpy as np

# Step 1: Generate ESM embeddings for all proteins in dataset
def embed_sequences(sequences):
    """Call ESM endpoint (from step 2) to batch embed sequences."""
    ACCESS_TOKEN = subprocess.check_output(["gcloud", "auth", "print-access-token"]).strip().decode()
    response = requests.post(
        f"https://us-central1-aiplatform.googleapis.com/v1/projects/{PROJECT}/locations/us-central1/endpoints/{ESM_ENDPOINT_ID}:predict",
        headers={"Authorization": f"Bearer {ACCESS_TOKEN}"},
        json={"instances": [{"sequence": s} for s in sequences]}
    )
    return np.array(response.json()["predictions"])

# Step 2: Create Matching Engine index
=======
conn.execute("SELECT * FROM alphafold_batch WHERE mean_plddt > 90 ORDER BY mean_plddt DESC")

5. Vertex AI Matching Engine for Protein Similarity Search

from google.cloud import aiplatform

>>>>>>> origin/main
index = aiplatform.MatchingEngineIndex.create_tree_ah_index(
    display_name="protein-embedding-index",
    contents_delta_uri=f"gs://{BUCKET}/protein-embeddings/",
    dimensions=1280,  # ESM3 embedding dimension
    approximate_neighbors_count=150,
    distance_measure_type="DOT_PRODUCT_DISTANCE",
)

<<<<<<< HEAD
# Step 3: Query for similar proteins
=======
>>>>>>> origin/main
endpoint = aiplatform.MatchingEngineIndexEndpoint.create(
    display_name="protein-similarity-endpoint",
    public_endpoint_enabled=True,
)
endpoint.deploy_index(index=index)
<<<<<<< HEAD

# Find proteins similar to query
query_embedding = embed_sequences(["MKTAYIAKQR..."])[0]
=======
>>>>>>> origin/main
neighbors = endpoint.find_neighbors(
    deployed_index_id=DEPLOYED_INDEX_ID,
    queries=[query_embedding.tolist()],
    num_neighbors=10,
)

<<<<<<< HEAD

6. Cost-Optimized Batch Docking via Vertex AI

Run DiffDock at scale using Vertex AI Batch Prediction with spot VMs:

from google.cloud import aiplatform

# Create batch prediction job (spot VMs = 60-90% cost reduction)
=======
## 6. Cost-Optimized Batch Docking via Vertex AI

```python
>>>>>>> origin/main
batch_job = aiplatform.BatchPredictionJob.create(
    job_display_name="diffdock-batch-screen",
    model_name=DIFFDOCK_MODEL_ID,
    instances_format="jsonl",
    gcs_source=f"gs://{BUCKET}/ligand-protein-pairs.jsonl",
    predictions_format="jsonl",
    gcs_destination_prefix=f"gs://{BUCKET}/docking-results/",
    machine_type="n1-standard-8",
    accelerator_type="NVIDIA_TESLA_T4",
    accelerator_count=1,
    starting_replica_count=10,
    max_replica_count=50,
<<<<<<< HEAD
    # Use spot VMs for 10-100x cost reduction
    service_account=SERVICE_ACCOUNT,
)

Gap Registry: What Vertex AI Cannot Replace

Related ASI Skills

• alphafold-database — structure retrieval; batch analysis feed for Vertex Pipelines
• esm — ESM3 for design; deployable as Vertex AI Endpoint
• diffdock — docking; deployable as Vertex Batch Prediction job
• deepchem — ADMET screening; post-design filter stage
• adaptyv — wet-lab validation; final pipeline stage
• torchdrug — KG reasoning; drug target validation
• uniprot-database — sequence/annotation retrieval; input data source
• gget — rapid queries; replaces manual API calls in pipeline
• bigquery-asi-interleave — parent GCP bridge; gnomAD query patterns
• vertex-asi-interleave — sibling Vertex bridge; generative AI patterns

• lolita — latent diffusion physics; analogous pipeline pattern for PDE emulation

  )

## Gap Registry

| Capability | ASI Skill | Vertex AI | Status |
|---|---|---|---|
| Structure retrieval (200M) | alphafold-database | Inference only | ASI owns retrieval |
| Protein sequence design | esm (ESM3) | Limited | ASI owns design |
| Molecular docking | diffdock | None | ASI owns this |
| ADMET prediction | deepchem | None | ASI owns this |
| Wet-lab validation | adaptyv | None | ASI owns this |
| BigQuery genomics | None | gnomAD, 1TB free | Vertex AI gap |
| Workflow orchestration | None | KFP Pipelines | Vertex AI gap |
| Scalable inference | Manual | Managed Endpoints | Vertex AI gap |
| Protein similarity search | None | Matching Engine | Vertex AI gap |
>>>>>>> origin/main