On-Device AI in Flutter & React Native: Running 3B Models Locally with WebGPU and ONNX Runtime

Sending sensitive user data (like health logs, personal note search, or voice recordings) to cloud LLMs creates privacy concerns and recurring API infrastructure costs.

On-Device Mobile AI enables cross-platform mobile applications (built with Flutter or React Native) to execute 1B to 3B parameter models directly on smartphone Neural Processing Units (NPUs) and GPUs.

1. Why On-Device AI for Mobile Apps?

• Zero Latency: Eliminates round-trip cellular network latency.
• 100% Offline Resilience: Applications remain fully functional in flight mode or low-connectivity zones.
• Complete Privacy: Zero user data leaves the physical device.

2. Integrating ONNX Runtime in Flutter

Using quantized ONNX models (.onnx format), cross-platform mobile apps leverage hardware acceleration via CoreML (iOS) and NNAPI / Vulkan (Android):

// Flutter ONNX Runtime Execution
import 'package:onnxruntime/onnxruntime.dart';

Future<List<double>> runOnDeviceInference(List<int> inputTokens) async {
  final session = await OrtSession.fromFile('assets/models/llama_3b_q4.onnx');
  final inputOrt = OrtValueTensor.createTensorWithDataList(inputTokens);
  
  final outputs = await session.run({'input_ids': inputOrt});
  return outputs.first.value as List<double>;
}

On-device AI unlocks next-generation mobile user experiences without relying on cloud infrastructure.