Local AI without pretending the cloud does not exist

“Local” and “cloud” are often presented as opposing camps. In practice, the useful architecture is usually a boundary between them.

The device is a good place for private context, low-latency interaction and work that should remain available without a connection. The cloud is good at coordination, distribution and tasks that need a stable public endpoint. Problems begin when an application sends everything away simply because the API is convenient.

Start with the sensitive path

I begin by tracing the information a feature touches. Which inputs reveal something about the user? Which intermediate results are just as sensitive as the original data? Does the final result need to leave the machine at all?

This often produces a smaller local component than expected. It does not need to reimplement the whole application. It needs to own the private path.

Treat the local runtime as a real dependency

Local inference is not automatically reliable. Machines differ, models may be missing and memory is finite. A good interface reports capabilities instead of assuming them.

The cloud-facing application should be able to ask:

• which operations are available;
• which model is ready;
• what resource limits apply;
• whether the operation can be cancelled.

That contract matters more than the transport underneath it.

Make failure understandable

If the local component is offline, the application should not silently send the same data somewhere else. A privacy boundary that disappears during an error is not a boundary.

I would rather show a clear explanation and let the user decide whether a remote fallback is acceptable. Trust is built in these unglamorous moments.

Hybrid should mean deliberate

The goal is not to keep every byte on one device. It is to make placement an architectural decision instead of an accident. Private thinking can stay local while the cloud handles the pieces it is actually good at.

That balance gives users something better than a slogan: control they can notice.