The Cheapest Inference Call Is the One You Never Make

Cutting frontier vision model costs with edge-side input gating

Most writing about LLM cost optimization — including my own — operates on requests you have already decided to make. Cache the prompt prefix. Route easy queries to a smaller model. Batch what isn't latency-sensitive. Distill a student for the hot path. All of it works, and all of it assumes the request itself is worth making.

Multimodal changes that assumption, because the arithmetic changes. Text requests are cheap enough that admission control isn't worth the engineering. Images aren't, and video is in a different category entirely. Once your pipeline is ingesting camera input, the largest cost lever moves upstream of the model call: deciding which inputs deserve one at all.

This is the layer above routing. Routing asks which model should handle this? Gating asks should any model handle this?

The arithmetic that forces the issue

Exact tokenization varies by provider and changes often enough that you should check current documentation rather than trusting a blog post, but the order of magnitude is stable: a single high-resolution image costs roughly as much as one to two thousand text tokens. Providers either tile the image into fixed-size patches at a fixed cost per tile, or approximate cost as a function of pixel area.

Apply that to video. Ten seconds at 30fps is 300 frames. At ~1,500 tokens each, that's 450,000 input tokens for ten seconds of footage — before any output, and before you multiply by users, sessions, and retries.

Nobody actually sends every frame, which means every production vision pipeline already implements gating. The question is only whether it does so deliberately. Uniform sampling — "take every tenth frame" — is gating with the intelligence turned off. It discards useful frames and forwards useless ones at exactly the same rate.

The two things worth filtering are different problems and need different mechanisms:

• Corruption. The input is degraded past the point of yielding a reliable answer. Blur, blown exposure, occlusion, subject out of frame.
• Redundancy. The input is fine but tells you nothing new. Frame 47 is nearly identical to frame 46.

In most pipelines redundancy is the larger cost, and it gets less attention because it doesn't look like a defect.

A cascade, not a filter

Each tier is roughly an order of magnitude cheaper than the next. The dashed path is the part most teams skip — without it, you can't tell a cost saving from a silent capability regression.

The pattern that works is a tiered cascade where each tier is one to two orders of magnitude cheaper than the next, and each tier's job is to reject only what the next tier demonstrably can't use. Cheap and certain first.

Tier 0 — metadata and signal processing. Microseconds, no model, no meaningful battery cost.

• Sharpness from variance-of-Laplacian or Tenengrad gradient energy
• Exposure from histogram clipping percentages and mean luma
• Device motion straight off the gyroscope and accelerometer — the IMU is already streaming on every phone and almost nobody uses it for this
• Global motion via sparse optical flow
• Near-duplicate rejection via perceptual hashing
• Actual versus nominal frame rate from capture metadata, which diverge more than people expect in low light

Build this tier first. It is nearly free, it needs no training data, and in most pipelines it removes a meaningful fraction of traffic on its own.

Tier 1 — a small on-device model. Single-digit milliseconds on a modern NPU.

A quantized MobileNetV3-Small or EfficientNet-Lite0 backbone, int8, under 5MB, with a multi-head output: a usable/unusable score plus reason codes, so the application can tell the user what to fix rather than just rejecting silently. Often you don't need a dedicated model at all — if you're already running an on-device detector or encoder for other reasons, its confidence scores, bounding boxes, and subject-size estimates are quality signals you're getting for free.

Platform-native detectors (Apple's Vision framework, Android's ML Kit) are worth serious consideration here because they ship zero bytes in your app bundle and cost nothing per call.

Tier 2 — a cheap cloud model. For inputs that pass the edge but don't clearly warrant frontier-class reasoning. This is conventional model routing, and it's well covered elsewhere.

Tier 3 — the frontier model. Now handling a small, pre-qualified fraction of original volume.

The technique worth stealing: distill the expensive model's own confidence

The hardest part of building a quality gate is defining quality. Hand-authored thresholds encode your assumptions about what matters, and your assumptions are worse than the data's.

There's a cleaner formulation. Your expensive model already tells you, implicitly or explicitly, when it couldn't do anything useful with an input — via low confidence, a refusal, a hedge, a downstream validation failure, or a user retry. That signal is a label. Log it.

Then train the tiny edge model to predict it: would the frontier model have produced a usable answer for this input?

Three properties make this attractive. The labels arrive free at production volume, so annotation cost is near zero. The objective is exactly the business objective — don't spend on inputs that won't pay off — rather than a proxy for it. And the gate improves automatically as the expensive model changes, provided you retrain on recent data.

The corollary is that you should not reach for an off-the-shelf no-reference image quality model like BRISQUE or MUSIQ and call it a gate. Those are trained to predict human perceptual judgment, which is a different construct from task utility. A sharp, well-exposed, correctly-framed image can be completely useless for your task, and a perceptual IQA model will wave it straight through.

Redundancy: semantic deduplication

Perceptual hashing catches near-identical frames. It doesn't catch frames that are visually different but semantically equivalent.

The approach that scales: run a small on-device image encoder — MobileCLIP-class, a few hundred KB to a few MB — over candidate frames, and forward a frame only when its embedding exceeds a distance threshold from the last frame you forwarded. You're sampling in embedding space rather than in time.

This adapts automatically to content rate. Static scenes collapse to a handful of frames. Rapidly changing scenes keep their density. Uniform temporal sampling can't do either, and in long-session video this single change routinely outperforms all your quality filtering combined.

Four ways this goes wrong

1. Naive thresholds discard your most informative inputs.

This is the failure I'd most want to warn people about, because it's systematic rather than random. In many domains the "degraded" inputs are the high-information ones. Fast motion produces blur — that's physics, not a defect — and the moment of fastest motion is frequently the moment you care about. Anomalies look like noise. Transitions look unstable. If you threshold each frame independently, you will reliably delete the peaks and keep the boring middle.

The fix is to make quality a clip-level and phase-level judgment rather than a per-frame one. Score frames, segment the sequence, and require the informative segments to be adequate — instead of requiring every frame to be pristine.

2. False rejects are invisible.

A gate is a classifier with badly asymmetric and asymmetrically observable errors. A false accept wastes a model call, which shows up in your bill. A false reject silently removes capability: the answer you'd have gotten never existed, so nothing appears in any dashboard. Your cost graph looks great. Your product quietly got worse.

The only reliable defense is a shadow stream. Sample some small percentage of rejected inputs and send them to the expensive model anyway, purely to measure what the gate is throwing away. Treat that as permanent instrumentation, not a launch-week experiment. If you build only one thing from this article, build this.

3. Cascade drift.

The gate was trained on the input distribution you had. New device generation, new camera defaults, new user population, new use case — and the gate's calibration decays while everything upstream looks fine. Monitor admission rate by device model, app version, and cohort. A sudden shift in admission rate is a leading indicator of something you haven't noticed yet.

4. Optimizing the wrong denominator.

If the metric is cost per API call, gating always wins by construction. The metric has to be cost per successfully resolved request. Cutting spend 70% while success rate falls 15% can easily be net negative once you price support load and churn.

Mobile-specific considerations

Everything above changes character when the first tier runs on a phone.

Thermal and battery are real constraints. Sustained NPU load throttles the device, and throttling arrives precisely during long sessions — the sessions where your heaviest users are most engaged. Duty-cycle the checks. Capture conditions change slowly; running a gate at 5–10Hz instead of per-frame at 60 gives up almost nothing and costs a fraction of the power.

App bundle size is a product metric. Every megabyte of model weight measurably affects install conversion. This is the strongest argument for platform-native detectors where they suffice, and for aggressive quantization where they don't.

Runtime choice is a maintenance decision, not a benchmark decision. Core ML on iOS and LiteRT (with QNN or vendor delegates) on Android give the best per-platform performance; ExecuTorch or ONNX Runtime Mobile give you one codebase across both. Pick based on how many platform-specific pipelines your team can actually maintain.

Bandwidth is part of the cost you're saving. Inference spend gets the attention, but not uploading the input at all saves egress, saves the user's cellular data, and removes upload latency from the critical path. On slow connections the latency win can exceed the inference win from the user's perspective.

Degraded connectivity becomes a feature. Once the first tier is local, the app can give immediate feedback with no network at all, and queue qualified inputs for later. A product that stays responsive offline reads as well-built rather than as broken.

The privacy dividend is free positioning. Inputs rejected at the edge never leave the device. In regulated contexts, and increasingly in unregulated ones, "we don't upload what we don't need" is worth more than the compute it saves.

Timing beats triage. Guidance offered before capture is far more valuable than filtering after it. Rejecting an input the user can no longer re-create is a cost saving and a retention problem at the same time.

Instrument these five things

• Admission rate per tier, segmented by device, app version, and user cohort
• Cost per resolved request — the headline number, and the only one that can't be gamed
• False reject rate, from the shadow stream, with a fixed alert threshold
• Escalation rate from cheap cloud model to frontier model, which tells you whether Tier 2 is pulling its weight
• p95 end-to-end latency, because cascades add hops and the tail is where users notice

When not to bother

Gating is infrastructure, and infrastructure has a fixed cost that has to be amortized.

Skip it if your volume is low enough that a quarter of engineering time exceeds the projected savings. Skip it if a missed input is catastrophic rather than merely annoying — some domains should send everything and eat the cost. Skip it if your inputs are already curated by a human, since you'd be re-deriving a judgment that's already been made. And don't build the sophisticated tiers before the free ones: Tier 0 signal processing plus honest measurement will tell you whether Tier 1 is worth training, and sometimes the answer is no.

The shape of the idea

Caching, routing, batching, and distillation all make the call cheaper. Gating questions whether the call should happen. For text pipelines that distinction is mostly academic. For anything ingesting images or video, it's where most of the recoverable money is — and it's upstream of every other optimization, which means it compounds with all of them.

The discipline it requires is unusual, though, and worth stating plainly: you have to invest in measuring what you're not doing. A gate you can't audit is not a cost optimization. It's a capability regression that hasn't been discovered yet.