Small models on the edge - what can 3B parameters actually do now?

A 3B model running locally on a laptop now handles summarization and tool calls that needed a datacenter model two years ago. The gap is closing fast for narrow tasks. Where have you successfully replaced a frontier API call with a small local model, and where did it fall apart?

On ARM Linux edge boxes we have replaced a frontier API for a few narrow jobs, and it falls apart as soon as the task leaves that envelope.

What has worked: short summarization of local logs or tickets, extracting a fixed JSON schema, and simple tool routing (pick one of N local commands, fill arguments, refuse if uncertain). A 3B-class quantized model, with a tight system prompt and constrained decoding, is fast enough on a CPU or a small NPU, stays offline, and does not need a datacenter round trip. We measure tokens per second, peak RSS, and thermal throttling, not only quality.

Where it broke: multi-step tool use with weak schema discipline, long-context synthesis across many documents, and anything that needs up-to-date facts. The model will still produce a fluent answer, but the argument types or citations drift, and there is no cheap way to see it without a verifier. We kept a small cloud model behind a policy for those paths, and only let the local 3B handle tasks that have a validator or a human in the loop.

So the gap is closing for narrow, checkable work. I would not treat “it can summarize and call tools” as a general replacement until the tool contract and the failure mode are as explicit as the model size.