For years now, running an AI model on an ESP32 generally meant a keyword spotter, a vibration classifier, or perhaps a model that decides whether your security camera saw a cat or a raccoon. That's TinyML, and it's genuinely useful, but it's not what you'd expect out of an LLM.
I'd mostly accepted that a microcontroller running anything resembling an LLM was a novelty at best, nothing more than a demonstration. Then someone managed to run a 28.9-million-parameter model on this $8 chip, and suddenly the impossible looked very much possible.
Where the bar used to sit
260,000 parameters, and every one earned its keep
Back in 2024, a developer named Dave Bennett got a real language model running on an ESP32-S3, using a 260,000-parameter checkpoint trained on the TinyStories dataset and Andrej Karpathy's llama2.c as the runtime. He picked a board with 2MB of PSRAM, since even that tiny model needed about 1MB of RAM for its weights.
Leaning on the chip's SIMD instructions, both cores got the model running at about 19 tokens per second. It worked fine, wrote little stories, but it represented something close to a ceiling. Every parameter had to live somewhere fast, and 512KB of SRAM paired with a couple megabytes of PSRAM capped how big it could get.
Someone moved the goalposts
A hundredfold jump in parameters, and zero extra RAM
A developer going by slvDev shipped a project that runs a 28.9-million-parameter model on the same class of $8 chip, at around 9.5 tokens per second, with nothing sent to a server. That's roughly a hundred times more parameters than Bennett's model, on similar hardware.
What changed is the assumption that every parameter needs to be fast. Most of a language model's weights sit in an embedding table it reads from rather than computes with, and you usually only need the rows tied to the current token. So instead of cramming 25 million of those parameters into the SRAM, the project leaves them in flash, memory-mapped, and pulls about six rows, roughly 450 bytes, per token.
The part that does the actual math in every step, the dense core, is a tiny 559,000 parameters that fit easily in fast memory. Google calls this idea per-layer embeddings, and it's part of how Gemma 3n and Gemma 4 run efficiently on phones. It's the same trick behind Google's offline translator project, just adapted to run on a microcontroller's memory layout instead.
It's not just a neat shell trick either. The project author tested it with a plain baseline, the full lookup table, a simpler alternative spending the same parameter budget differently, and a control wired identically but without a real table behind it. The full table cut perplexity, a measure of prediction quality, by about 9 percent over baseline, while the control did worse than nothing. The table earns its keep, and the advantage survives the drop to 4-bit quantization needed to fit it in flash.
Getting it onto my own board
Half the flash, a smaller dictionary, same trick
My own board isn't the N16R8 the project targets; it's an ESP32-S3 N8R8 with half the flash and none to spare. The 14.9MB stock model needs 16MB just to hold firmware and weights together, so the published build doesn't fit my board as is.
In order to get it working, I had to shrink the vocabulary first, since a smaller vocabulary means a smaller lookup table and output head. I retrained a reduced-vocabulary version with the project's own training pipeline, resized the partition table for 8MB flash, and recompiled the firmware with the matching flash-size flag before writing the new model to the same custom partition.
It booted, and the same three-tier split held: dense core in SRAM, output head in PSRAM, lookup table read a few hundred bytes at a time from the flash. I ended up getting 22.3 token/second on my ESP32-S3, the 4096-word vocabulary makes the output head about six times cheaper than the stock 32,768-word model, so this build is actually faster The technique doesn't need the biggest board to prove the point, just a memory hierarchy worth respecting.
There's still a catch
The wall moved; it didn't vanish
None of this makes the chip any faster than it already is, and you're still limited by the hardware on the humble ESP32. Profiling shows more than half of each token's 105-millisecond budget goes to the output head in PSRAM, bottlenecked by scan speed, not flash reads. The 25-million-parameter table everyone worries about only costs about 8.5 milliseconds. So the bottleneck simply moves from not having enough memory to not having enough memory bandwidth — a much better problem to have.
The model itself is still a novelty, and nowhere near what local LLMs can do on an old laptop, PC, or even Raspberry Pi today. Trained only on TinyStories, it writes short, coherent little tales and nothing else. No facts, no reasoning, only vocabulary and style.
The ESP32 strikes again
This little board just keeps going on
The honest scope of the project is still quite limited. It's not a chatbot in your pocket, but it is proof that a microcontroller can hold a hundred times more model than we assumed, as long as you're willing to stop treating flash as a last resort.
I didn't think an ESP32 could run something you'd call an LLM with a straight face. Turns out I was right about the chip, but wrong about where the limit actually is.