Apple On Device AI

On-Device AI for Apple Platforms

Architect and deploy private on-device machine learning and LLMs on Apple Silicon. Covers framework selection between Apple Foundation Models, Core ML, MLX Swift, and llama.cpp, plus model optimization and quantization.

Scope Boundary: Swift-side Core ML inference and model caching live in coreml. This skill owns backend selection, architecture, model conversion, and LLM runtimes.

Contents

• Framework Selection Router
• Apple Foundation Models
• Core ML & Model Conversion
• MLX Swift & Open Weights
• Model Compression & Optimization
• Common Mistakes
• Review Checklist
• References

Framework Selection Router

Apple Foundation Models

Leverage system-provided generative models on iOS 26+ without bundling model weights:

import FoundationModels

let session = LanguageModelSession()
let response = try await session.respond(to: "Summarize today's highlights in 3 bullet points.")
print(response.text)

Use @Generable to extract structured data directly from model prompts.

Core ML & Model Conversion

Convert PyTorch and Hugging Face models using Python coremltools:

import coremltools as ct
import torch

model = MyPyTorchModel().eval()
traced = torch.jit.trace(model, torch.randn(1, 3, 224, 224))

mlmodel = ct.convert(
    traced,
    inputs=[ct.TensorType(name="input", shape=(1, 3, 224, 224))],
    compute_units=ct.ComputeUnit.ALL
)
mlmodel.save("Model.mlpackage")

MLX Swift & Open Weights

Run open-source LLMs leveraging Apple Silicon unified memory:

import MLXLLM

let model = try await LLMModelFactory.shared.load(modelName: "mlx-community/Llama-3.2-3B-Instruct-4bit")
let output = try await model.generate(prompt: "Explain relativity in simple terms.")

Model Compression & Optimization

• Quantization: Compress weights to 4-bit (int4) or 8-bit (int8) via coremltools.optimize.coreml to reduce memory bandwidth bottlenecks.
• Palettization: Cluster weights into lookup tables for smaller download sizes.
• Pruning: Zero out non-critical weights to accelerate inference.

Common Mistakes

• Bundling massive LLMs in app bundles: Causes app review rejection or slow downloads. Use Background Assets or system Foundation Models.
• Ignoring device memory limits: Loading a 4-bit 7B LLM requires >4GB RAM, risking immediate OOM termination on devices with 6GB or 8GB unified memory.
• Running autoregressive LLMs without KV-cache: Re-evaluating the full token prompt at each step slows generation exponentially.
• Neglecting thermal throttling: Sustained GPU/CPU inference causes thermal throttling within minutes. Leverage the Neural Engine for sustained workloads.

Review Checklist

• Appropriate framework selected for latency, power, and storage constraints
• System Foundation Models prioritized when Apple Intelligence features suffice
• Memory footprint measured on the lowest supported physical device target
• Models quantized to 4-bit or 8-bit to fit within memory budgets
• Neural Engine dispatch verified using MLComputePlan or Instruments

References

• Foundation Models API -- LanguageModelSession, @Generable, tool calling, prompt design
• Core ML Conversion -- Model conversion from PyTorch, TensorFlow, other frameworks
• Core ML Optimization -- Quantization, palettization, pruning, performance tuning
• MLX Swift & llama.cpp -- MLX Swift patterns, llama.cpp integration, memory management