The shrinking giants: why tiny AI models like Needle matter for everyone

For the past three years, the story of artificial intelligence has been a story of scale. Bigger models, bigger training runs, bigger data centres, bigger electricity bills. The headline-grabbing systems from OpenAI, Google, Anthropic and Meta have ballooned into the hundreds of billions of parameters, and the cost of running them has ballooned with them.

So when a developer turns up on Hacker News with a project called Needle — a 26-million-parameter model distilled from Google's Gemini that can reportedly handle tool calling — it's worth paying attention. Not because Needle is going to replace GPT-5 or Gemini Pro tomorrow, but because it's another data point in a quiet counter-trend: small, focused models are becoming genuinely useful, and that has real consequences for how AI gets built, deployed and paid for.

What Needle actually is (and why 26 million is a small number)

To put 26 million parameters in perspective: GPT-3, released in 2020, had 175 billion. Modern frontier models are widely estimated to be far larger still. A 26M model is roughly four orders of magnitude smaller than the systems most people think of when they hear "AI". It's small enough to fit comfortably on a phone, a Raspberry Pi, or a modest laptop CPU — no GPU required.

Needle's claim to fame is "tool calling" — the ability of a language model to decide when to invoke an external function (search the web, query a database, send an email, run a calculator) and to format the request correctly. Tool calling is one of the workhorse capabilities behind modern AI agents and assistants. Historically, doing it reliably has been the preserve of large frontier models, because the model needs to understand intent, pick the right tool from a menu, and produce a structured output.

The technique Needle uses to punch above its weight is knowledge distillation: a small "student" model is trained to mimic the outputs of a large "teacher" model — in this case, Gemini. Rather than learning from raw text, the student learns from the teacher's behaviour on a narrow task. For something as constrained as tool calling, where the universe of correct outputs is relatively bounded, distillation can yield enormous compression ratios.

The case for small, specialised models

The dominant narrative around AI has been that capability scales with size, and for general intelligence that's largely true. But most real-world deployments don't need general intelligence. They need a model that can do one thing — classify a support ticket, extract fields from an invoice, route a customer query, call the right API — and do it reliably, cheaply, and predictably.

Small models offer several concrete advantages here:

What this means for accessible AI

For Australian developers, startups and even hobbyists, the rise of capable small models is arguably the most democratising trend in AI right now. The economics of frontier models favour incumbents with billion-dollar capex budgets. The economics of distilled small models favour anyone with a laptop and a clear problem to solve.

Consider what this enables:

None of this is hypothetical. The open-source ecosystem around small models — Llama variants, Phi, Gemma, Mistral, Qwen, and now distilled niche models like Needle — has matured to the point where competent engineers can stand up production systems in days, not months.

The catch: small models are not magic

It's worth being honest about the limits. A 26M model distilled for tool calling is not going to write your essay, debug your codebase, or hold an open-ended conversation about philosophy. The whole point of distillation is to compress a narrow slice of capability, and the compression is only lossless within that slice.

This creates an architectural question that the industry is only beginning to grapple with: when should you reach for a small specialist, and when do you need a generalist? The likely answer is a hybrid — a fleet of cheap specialists handling 90% of requests, with a frontier model reserved for the genuinely hard cases. That's roughly how human organisations work, and it's how AI deployments are quietly starting to look behind the scenes.

There's also a dependency question. A model distilled from Gemini is, in a real sense, a derivative of Gemini. The terms of service around using frontier-model outputs to train competitors are murky, contested, and vary by provider. Anyone building commercial products on distilled models needs to read the fine print carefully.

A more plural AI landscape

The Needle project, and others like it, point toward a future that looks less like one or two godlike models running the world and more like a sprawling ecosystem of specialised tools, each tuned for a job. That's a healthier picture in almost every way: less concentration of power, more competition, lower costs, better privacy, less environmental impact, and a much lower barrier to entry for anyone who wants to build with AI.

It also reframes what "AI progress" means. For the past few years, progress has been measured almost entirely on the frontier — bigger benchmarks, bigger models, bigger claims. Distillation work, by contrast, is about making existing capabilities cheaper, smaller and more available. It's less photogenic, but arguably more consequential for the people who will actually use AI day to day.

A 26M-parameter model that can call tools is not going to make the front page of the international press. But for the developer in Brisbane trying to ship a useful product without taking on a six-figure infrastructure bill, it might be the more important story.

Related on Bleen

Sources