Why SQLite is quietly becoming the backbone of durable AI workflows

For most of the last decade, the default answer to "what database should I use?" was some flavour of managed Postgres, with a Redis queue bolted on for background jobs and a message bus for anything that smelled like a workflow. SQLite, the tiny embedded database shipped inside every browser, phone and aircraft on the planet, was filed away as something you used for unit tests or a hobby app.

That assumption is being quietly demolished. A wave of new platforms aimed at long-running AI agents and "durable execution" — code that can pause for minutes, hours or days and survive crashes without losing its state — is converging on the same unfashionable answer: just use SQLite. Cloudflare is putting a SQLite database next to every tenant in its serverless runtime. Google's Agent Development Kit (ADK) leans on durable session state so agents can resume mid-conversation. And the creator of the Obelisk workflow engine is making the case that, for orchestrating AI agents, SQLite straight-up beats cloud queues.

If you write software for a living, this shift is worth paying attention to. It changes the shape of the architecture diagram you'll be drawing in 2025.

What "durable" actually means here

The word getting thrown around in all of these announcements is durable. In plain English, a durable workflow is one where the state of a running program — which step it's on, what variables it has, what it's waiting for — is persisted continuously, so that if the process dies, the machine reboots, or a deployment rolls over, the workflow picks up exactly where it left off.

This matters enormously for AI agents. As Google explains in its post introducing long-running agents with ADK, real agents need to pause, resume, and never lose context — they might be waiting on a human approval, a slow tool call, or an external webhook that won't fire for hours. A traditional request/response web stack falls apart here, because nothing is designed to hold conversation state across days of wall-clock time.

The old answer was a heavy bit of plumbing: a workflow engine like Temporal or Airflow, a queue like SQS or RabbitMQ, and a relational database to keep track of what happened. The new answer being floated by several different teams independently is: skip most of that, and put a small, fast, transactional database right next to the code.

Cloudflare's bet: a SQLite per tenant

Cloudflare's recent announcements make the architectural shift extremely concrete. In Introducing Dynamic Workflows: durable execution that follows the tenant, the company describes a model where each customer's workflow state lives in a Durable Object — Cloudflare's name for a single-threaded, stateful compute primitive — that travels with the tenant rather than living in a centralised cluster.

The companion post, Durable Objects in Dynamic Workers: Give each AI-generated app its own database, takes this further. Every AI-generated application running on the platform gets its own embedded SQLite database, isolated from every other tenant. There's no shared Postgres cluster to scale, no connection pool to exhaust, no noisy neighbour to worry about. If you have ten thousand AI-generated micro-apps, you have ten thousand independent SQLite files, each one a few kilobytes until it needs to grow.

This is a meaningfully different shape from what most Australian engineering teams are used to. Instead of "one big database with tenant_id columns everywhere", it's "one database per tenant, full stop". The isolation story for compliance — think privacy obligations under the Australian Privacy Principles, or sectoral rules in health and finance — becomes much easier to explain to a security reviewer when each customer's data lives in a physically separate file.

SQLite vs the cloud queue

The most provocative pitch comes from the Obelisk engine. As Tech Times reports, Obelisk's creator argues that SQLite outperforms cloud queues for orchestrating AI agents. On its face that sounds absurd — SQLite is a single file on disk, while a managed queue is a fully distributed service.

But the argument is more subtle. Cloud queues are optimised for one thing: getting a message from producer A to consumer B as quickly and reliably as possible. They are not optimised for the bread-and-butter operations of a workflow engine: "show me every step this agent has executed", "resume this run from step 14", "find every run that's been waiting on a tool call for more than five minutes", "replay this run with different inputs". Those are queries, and queries are what relational databases are good at.

When you push agent state through a queue, you typically end up writing it to a database anyway, just to make any of the above possible. SQLite collapses the two layers. The workflow's events, its current step, its inputs and outputs all live in one transactional store, co-located with the code that reads and writes them. Latency for a local SQLite write is measured in microseconds, not the milliseconds you'd burn on a round trip to a managed queue.

Why now? Three things changed

SQLite isn't new — it shipped in 2000 and has been the most-deployed database in the world for years. So why is it suddenly showing up at the centre of serious infrastructure?

What developers should actually know

If you're building anything that smells like an agent, a background job runner, or a per-tenant app, SQLite has earned a spot on your shortlist. A few practical things to keep in mind:

The bigger picture

Architectures move in cycles. We spent the 2010s pulling state out of the application, into giant shared databases and dedicated services, in the name of scalability. The 2020s look like they're pushing some of that state back — not because shared databases are bad, but because compute has become so cheap and so granular that it's now reasonable to give every tenant, every agent, every AI-generated app its own little world.

SQLite is the database that fits naturally into that world. It's small enough to instantiate per user, fast enough to use as a workflow log, boring enough to trust with your data, and free enough that nobody's going to argue with the procurement bill. For Australian teams building agentic features without a hyperscaler-sized SRE budget, that combination is hard to beat.

The headline above is a bit cheeky — SQLite obviously isn't all you need. But for a surprisingly large class of modern workloads, it's the boring core the rest of your stack can be built around. That's a better place to start than most of us realised.

Related on Bleen

Sources