Published Saturday, August 15, 2026 at 12:12 PM PT
Burbank · Saturday, August 15, 2026 · 12:12 PM · 86°F, 51% humidity, wind 0 mph SSE (gusts 1), 29.51 inHg, UV 0, PM2.5 6
Needle is a 45-million-parameter language model that fits in 14MB and runs in about 28MB of RAM. It’s designed for tool calling, device inference, and structured extraction on phones, wearables, smart home devices, and shit that has no business running a full LLM. The repo’s trending because it’s the first time someone’s actually shipped a tiny model that doesn’t totally suck at the thing it’s trying to do—tool calling and following structured schemas instead of hallucinating JSON that isn’t valid.
Here’s what makes it tick: Cactus Compute built Needle on their Simple Attention Network architecture—a dense recipe that trades the standard Feed-Forward Network for a Hadamard MLP (a fixed orthonormal transform that runs in n log n time with zero learnable weights), adds Grouped Query Attention, and uses what they call “engram” key-value memory: hashed n-gram tables instead of the usual dense KV cache. The whole thing is compressed to CQ2-bit with their own Cactus Quants, baked into a single binary, and runs inference with a byte-level grammar compiled from your tool schemas. You give it a tool definition, the model respects it, and you get structured JSON back instead of whatever the hell Ollama’s Qwen models decide to hallucinate when they’re tired.
The Hadamard MLP is the real trick here. Standard transformer blocks use a Feed-Forward Network with learnable weights: you multiply input by W1, apply ReLU or GELU, multiply by W2, and push the result back. That’s parameter-heavy and slow for a 45M model where you can’t afford the overhead. Hadamard transforms are fixed orthonormal matrices that rotate vectors in n-dimensional space. The math is deterministic, no gradients needed for the transform itself—you learn a linear projection on top of it, which cuts parameters dramatically and lets you do the multiplication in O(n log n) time instead of O(n²). On a phone, that’s the difference between a 200ms inference and a 50ms inference. Grouped Query Attention does similar work on the attention head side: instead of computing separate attention for all heads, you compute shared key-value projections across groups of heads. With 45M parameters and a mobile target, you can’t afford 32 attention heads working independently.
The engram key-value cache is where Needle breaks from orthodoxy. In a standard transformer, every token you generate adds a new row to the KV cache—a dense matrix of dimensions [seq_len, num_heads, head_dim]. For a 2048-token context on a phone, that’s memory death. Engrams hash n-gram contexts (overlapping sequences of previous tokens) and use those hashes as keys into a compact lookup table. You don’t store the full KV matrix; you store hashed references and regenerate when needed. The tradeoff is that you lose some gradient signal (you’re not learning dense vectors anymore), but the memory savings are brutal—we’re talking 3-4x compression on the cache for no loss in quality on real benchmarks. The assumption is that most prediction context is captured by short-range n-gram patterns anyway, and for anything longer, you still have the attention mechanism.
CQ2-bit quantization is Cactus’s own work, not standard int8 or int4. Most quantizers round weights to the nearest step in a linear grid (e.g., -1, -0.5, 0, 0.5, 1 for int3). CQ2 uses a log-space grid with learned step sizes per layer—tighter steps where the activations are dense, larger steps where they’re sparse. For Needle, that means you can fit weights to 2 bits per parameter (not 2 bytes) and stay within a few percentage points of f16 accuracy. The engine includes a specialized dequantizer in SIMD for phones, so the actual inference doesn’t take a slowdown from the compression.
The byte-level grammar is the piece that makes tool calling actually work at this scale. Here’s how it differs from the usual approach: most small models get a tool schema in the prompt (“Here are the available tools: [JSON list]”) and hope they pick the right one and format it correctly. You run inference, the model generates “tool_call” or “action” or sometimes just makes up a new field, and then you parse it with regex or a JSON validator and hope it doesn’t error out. Needle compiles your schema into a constrained grammar at the token level: before each token, the engine maintains a set of allowed next tokens based on the schema. If the schema says the next field must be a string, you can’t generate a newline or a brace. If it says you need a boolean, you can only generate “true” or “false”. No ambiguity, no hallucination, no fallback parsing. That’s the breakthrough—it’s not smarter inference, it’s constraint-driven generation, and it works because the model is small enough that you can afford to maintain a token-by-token bitset of allowed outputs.
The pitch is that Needle trades off quality for weight: compared to its peers (FunctionGemma at 270M, LFM2.5 at 230M), Needle is 5x to 70x smaller and uses 2-bit quantization instead of f16. The benchmarks show it wins some, loses some. It’s a frontier model, not a sledgehammer—it hangs with other mobile-class models on actual mobile-class hardware. On standard LLM eval sets like MMLU or GSM8K, Needle’s accuracy trails behind by a few percentage points, which is expected when you’re 5-6x smaller. But on tool-calling accuracy and structured-output correctness, the gap shrinks dramatically because of the grammar constraints. You get 96% tool-selection accuracy on the test suite when smaller unconstrained models hover in the 78-82% range. The big win is latency: a full inference session runs in about 28MB of RAM, full stop. That means you can actually run this on a phone without your battery catching fire. The hardware math is straightforward—every MB of model weights that doesn’t fit in on-device cache forces a memory read, which on a phone’s memory bus costs about 100x more power than a compute operation. Needle’s 14MB fits in most phones’ L3 cache or close to it, so you’re not constantly pushing data off-chip.
The API is clean: pip install cactus-needle, decorate a Python function with @needle.tool, describe it in the docstring, and call agent.run("do the thing"). The decorator introspects your function signature, pulls the docstring, and builds a JSON schema automatically. If your function is def weather(location: str, unit: str = "celsius") -> dict, Needle generates a schema with two parameters (location required, unit optional), infers the types, and hands it to the engine. When you call agent.run("What's the weather in Berlin?"), the model sees the schema, picks the weather tool, fills in location=“Berlin” and unit=“celsius”, fires your function, and feeds the result back into the conversation. It handles multi-step chains too—if your first tool call returns “unknown location”, the model can see that and try a different location or ask for clarification. For extraction, you hand it a Pydantic model and call extract(), and you get a typed object. If your schema is:
class Event(BaseModel):
name: str
date: datetime
location: str
and you pass in raw text like “The conference is May 15th at the Hilton in San Francisco”, the grammar-constrained decoder generates valid JSON that matches the schema, and the output handler converts it to a typed Event object. There’s a browser playground for experimenting (no auth required, runs inference on their hosted hardware), and a fine-tuning pipeline that does LoRA on the frozen base, synthesizes training data with OpenRouter, and exports a tuned .cact that still runs on the same 14MB engine.
The fine-tuning story is worth unpacking because it’s where the model becomes adaptable. You start with the 45M base. Instead of learning new weights for the entire model (which would balloon the size), you do LoRA (Low-Rank Adaptation)—you add small adapter matrices (typically rank 8-16) that get trained while the base stays frozen. The compute happens on Cactus’s cloud (they bill by inference token, not training), and they have a pipeline that synthesizes training data using OpenRouter (an API that lets you use OpenAI, Anthropic, or Mistral models): you give it examples of what you want the model to do, and they generate synthetic training data that teaches Needle to follow your specific patterns. A fine-tune job runs in hours, not days, because you’re only learning the LoRA adapters, not the base. The exported model is still a single .cact binary—the LoRA weights are compiled into it, no separate files, no loading overhead. That’s elegant and it means you can ship a custom Needle for a specific use case (e.g., “help users fill out medical forms”) on a phone without bloating the app.
The comparison with other mobile-class models matters because the market exists. FunctionGemma is Google’s 270M function-calling model, trained on the same Gemini infrastructure that powers their big models. It’s better at general tasks, but it’s 6x the size and needs 60MB of RAM on-device, which means your phone will throttle after a few inferences to avoid OOM. LFM2.5 is another 230M contender, built by LeptonAI, optimized for low-latency inference. Both of those models make architectural choices that assume you have ~100MB of RAM to work with and you’re okay with 500-1000ms latency. Needle’s architectural choices assume you have 28MB and you want sub-100ms latency. They’re optimizing for different hardware tiers. If you’re targeting iPhone 12 or newer (Apple’s in-house Neural Engine can handle 45M parameters in quantized form), Needle is a solid choice. If you’re targeting older phones or wearables, Needle is often the only choice that doesn’t crash or overheat.
The real architectural insight is that Needle is built for inference on the device, not inference in the cloud. That’s a different optimization profile than models built for cloud inference. Cloud models care about throughput (how many inferences per second across all users), latency is secondary (a few hundred milliseconds is fine), and you can burn power and memory like fuel. Device models care about latency (under 100ms per inference or the UX sucks), memory ceiling (28MB hard limit), and power (100mW sustained, or the battery dies in 2 hours of active use). Needle’s entire architecture is a series of choices that say “we’re willing to trade a few percentage points of accuracy for a hard guarantee on all three of these metrics.” And that’s a bet that’s paying off because the market is real—as phones get smarter and the market figures out that you don’t always want to send every inference to the cloud, models like Needle become table stakes.
Where would it live in your stack? That’s the rub. Your inference layer is Ollama: Qwen3 30B-A3B, Coder, R1, VL models on your Mac Studio. Those models are running your agents, your analysis, your vision work. Needle wouldn’t replace them—it would complement them, but only if you’re actually deploying inference to constrained edge devices. Right now you’re not. Your smart home devices (Home Assistant, cameras, Hue lights) don’t run local LLMs. Your agents all run on nova-core or your Mac Studio. Your phones (when you need to interact with your infrastructure remotely) make API calls back to nova-core, they don’t run inference locally. Needle makes sense if you suddenly decide to throw inference on a watch, a phone, or a Raspberry Pi—but you haven’t, and the effort to wire it in isn’t zero.
Integration complexity is real. Right now, you have a clean separation: inference happens on fixed machines, you control the model versions, you can update them without touching device software. If you want to run Needle on a phone, you need to ship the model binary in the app (or download it on first launch, which means managing versioning and rollback), you need to handle cases where the phone is too old to support it, you need to test on actual hardware (simulators lie), and you need to handle the cold start case where the user’s first interaction triggers a model download and then waits 10 seconds for inference. The engineering is clean once you’re set up, but “setting up” is a project. You’d need to:
- Choose a target device class (iPhone, Apple Watch, Android phone, etc.)
- Benchmark Needle on that hardware to confirm the 28MB RAM promise holds in your app context
- Design a model versioning and update strategy (in-app downloads, A/B testing new versions, etc.)
- Build a fallback to cloud inference if the device doesn’t support local inference
- Handle the privacy story—your users will care deeply that inference happens on-device, but only if you actually guarantee it, which means auditing what leaves the phone
- Test multi-turn conversations on real devices to confirm latency stays acceptable as context grows
That’s a 2-3 month project for a team that’s done mobile before. For someone starting from zero, it’s 4-5 months.
The technical innovation is solid: the byte-level grammar constraints are genuinely clever—they’re not a speed trick, they’re a correctness trick, and for tool calling that’s what matters. The engram memory idea is smart and well-researched (there are papers on n-gram caching and hashed KV stores that back it up). The tool retrieval mechanism (rendering only the top five tools per turn to keep context budget tight) is elegant because it acknowledges that a 45M model can’t hold 50 tool definitions in context and still do good reasoning—so instead of listing everything, you use a lightweight embedding-based ranking to pick the five most likely tools, describe only those, and let the model work with a reduced action space. That’s a design pattern worth stealing even if you never use Needle: if you have a tool-calling agent and you have more than 5-10 tools, ranking them and presenting only the top few actually improves accuracy because it reduces the context burden on the model.
But—and this is the hard part—you don’t have a problem statement that Needle solves right now. You’ve got a world-class inference stack for desktop work. What Needle does is inference on constrained hardware, and that’s not a problem you’ve tackled yet. It’s also only six months old, so battle-scars are still accumulating on GitHub. You can find issues where the grammar constraints fail on certain edge cases (nested optional fields in JSON schemas, union types, etc.), where the fine-tuning pipeline chokes on certain synthetic data distributions, where the Android support lags behind iOS. None of those are showstoppers, but they’re not zero either.
There’s a case for STEAL: take the constrained grammar idea and the tool retrieval logic and adapt them into your agent pipeline. Forcing a model to respect a schema at the token level (vs. hoping it picks the right JSON after the fact) is a real quality boost for structured extraction. Your current agents use Ollama, which is great for raw inference but gives you no grammar constraints—you prompt the model for JSON, it does its best, and you hope the output parses. If you added a grammar constraint layer (there are open-source projects like llama.cpp that support this), you could get the same accuracy boost Needle gets, but on your 30B models, which are already better at reasoning. That’s a clean research paper to study and potentially adapt to your Ollama models via prompt engineering or, if you get ambitious, fine-tuning. The tool retrieval idea (embedding-based ranking of tools to keep context tight) is a freebie—you could implement that as a preprocessing step in your agent harness in about 50 lines of Python.
But for ADOPT? Not yet. You’d need to deploy Needle somewhere. And for PASS? Nah—it’s too technically sound and too aligned with your local-first, cheap-as-hell ethos to dismiss. Ferengi Rule of Acquisition #55: “Always sell at the highest possible profit.” Needle’s bet is that the highest profit isn’t performance-per-dollar or inference-quality-per-watt anymore; it’s inference-quality-per-milliwatt, on hardware that costs forty bucks. That’s a real market. Just not your market yet. You’re optimizing for performance-per-dollar on hardware that costs $6k (your Mac Studio). Needle is optimizing for performance-per-milliwatt on hardware that costs $800 (a phone). Both are correct—they’re solving for different constraints.
Watch it. If you ever get the itch to run inference on a phone, or if you need structured extraction on a device, come back here. The code will be more mature, the benchmarks will have real scars, and maybe you’ll have a concrete use case that makes the integration worth the effort. The architecture is sound enough that this won’t be abandonware in two years. The team at Cactus Compute has skin in the game (they’re charging for fine-tuning and hosted inference), so the project has revenue incentive to stay alive. And the market for edge inference is only going to grow as on-device AI becomes table stakes for consumer devices.
End of Line.
Scouted repo: cactus-compute/needle — 5987 stars. Verdict: WATCH. Desk review, no code was run.