denisovav The ONNX engine
Rules
- Depend on
flutter_gemmaandflutter_gemma_onnx, and import both. The engine package does not re-export core. - Declare
fileType: ModelFileType.onnx. Without it the install defaults totaskand never reachesOnnxEngine. - An ORT-GenAI model is a directory —
genai_config.json, the.onnxgraph, its weights and a tokenizer. Install it withfromHuggingFace(repo), which downloads the whole folder, or pointfromFileat a localgenai_config.json. A single-file download or a Flutter asset cannot produce it. - Native generation runs on macOS arm64, Linux x64, Windows x64, Android arm64 and iOS arm64. On any other native host no engine accepts the model and
getActiveModelthrowsNo inference engine can handle this model. Web is a separate arm with its own rules (below). - Android needs
minSdk 24— the build hook fails the build below it. Phi-3.5-mini peaks near 3.7 GB of RAM, so target 8 GB devices. - Text only: no images, no audio, no LoRA.
Setup
flutter pub add flutter_gemma flutter_gemma_onnx
import 'package:flutter_gemma/flutter_gemma.dart';
import 'package:flutter_gemma_onnx/flutter_gemma_onnx.dart';
await FlutterGemma.initialize(
inferenceEngines: [OnnxEngine()],
embeddingBackends: [OnnxEmbeddingBackend()],
);
await FlutterGemma.installModel(
modelType: ModelType.phi,
fileType: ModelFileType.onnx,
).fromHuggingFace('microsoft/Phi-3.5-mini-instruct-onnx').install();
final InferenceModel model = await FlutterGemma.getActiveModel(maxTokens: 4096);
A repo with several execution-provider folders resolves to its CPU/mobile folder automatically — the bundled runtime is CPU-only.
A bundle shipped with the app:
await FlutterGemma.installModel(
modelType: ModelType.phi,
fileType: ModelFileType.onnx,
).fromFile('$path/genai_config.json').install();
Sessions, chats and streaming work as in the flutter-gemma-inference skill, except openSession / openChat, which throw UnsupportedError here — one conversation at a time. Pass modelType to createChat for function calling; ONNX falls back to ModelType.gemmaIt.
Web
On web OnnxEngine runs the model through Transformers.js. Install it by Hugging Face repo URL; the browser downloads and caches the files on first use:
await FlutterGemma.installModel(
modelType: ModelType.qwen,
fileType: ModelFileType.onnx,
).fromNetwork('https://huggingface.co/onnx-community/Qwen2.5-0.5B-Instruct').install();
The repo must be in Transformers.js layout, as the onnx-community ones are. ORT-GenAI repos such as microsoft/Phi-3.5-mini-instruct-onnx do not run in the browser. fromFile and fromAsset install on web without complaint and then throw UnsupportedError from the first createSession — on web the model identity has to be a repo, not a file. fromBundled('<id>') serves one from the app's own origin. PreferredBackend.cpu forces WASM; anything else tries WebGPU first.
Add to web/index.html <head>, before Flutter boots — the first script for generation, the second for embeddings:
<script type="module">
window.transformersReady = (async () => {
const m = await import('https://cdn.jsdelivr.net/npm/@huggingface/transformers@4.2.0');
window.transformers = m;
return m;
})();
</script>
<script type="module">
window.ortReady = (async () => {
const m = await import('https://cdn.jsdelivr.net/npm/onnxruntime-web@1.27.0/dist/ort.bundle.min.mjs');
m.env.wasm.wasmPaths = 'https://cdn.jsdelivr.net/npm/onnxruntime-web@1.27.0/dist/';
window.ort = m;
return m;
})();
</script>
Embeddings
OnnxEmbeddingBackend handles single-file .onnx or .ort embedding models, installed with FlutterGemma.installEmbedder() like any other — see the flutter-gemma-rag skill for the indexing flow.