← all entries

Earl Grey ... Hot

November 15, 2026

In Star Trek IV: The Voyage Home, released in 1986, Scotty is brought to a 1980s Macintosh and asked to make some calculations. He picks up the mouse, holds it to his mouth, and says "Hello, computer." Nothing happens. A helpful engineer tells him to use the keyboard. Scotty looks at it and says, "Keyboard. How quaint."

The joke works because the audience already knew which interface was coming. The film was a ten-year bet that people would one day talk to machines, and the bet was right about the direction and wrong about the plumbing. Scotty never asks the computer to think. He asks it to do the thing he'd have typed, only in his own words.

That's the job to be done. A web app has some controls. A person says what they want, and something turns the sentence into control presses. The something can be small, and it has now been built twice.

A sentence in, a list out

The first version is a proof of concept called dowhat. You type put a camel in the lower right, and have it connect to gravity and repel other camels, and a Matter.js page shows a camel appear in the lower right, falling, pushing other camels away. The scene is a 0 to 100 grid. Objects have a position, a gravity switch, a repel switch, and can be wired to each other with a rope, chain, rod or spring.

Underneath, the sentence becomes a list of commands, and the list is the product. For that sentence it's three: spawn a camel at a region, turn its gravity on, turn its repel on. The same list format covers fifteen kinds of command in all, among them remove, move, nudge, flip, resize, trigger, and connecting or disconnecting two objects. That's the whole vocabulary, and the physics engine never sees anything that isn't in it.

Then, when the same interface is ported over to the scene engine that runs inside art-synthetic, the commands materialize drawings in and out of a live scene and change how they behave. The model-driven parser there is a back-port of the one in dowhat.

Grammar first, then the model

The first parser had no model in it. It was a spaCy grammar: tag the region words, walk the tokens by part of speech, and build the command list. It was ported from art-synthetic's own parser and checked against that project's corpus of real phrases. It works, and it fails the way a grammar fails, on the sentence nobody anticipated.

In September the live endpoint switched to a local model, Llama 3.2 at 3 billion parameters, running under Ollama on the same machine. The swap was one function. Both parsers produce the same intermediate form, a typed intent record, and everything after it (the compiler, the scene state, the reply text, the web API and the page) didn't know which parser had run. The grammar is still in the repo, tests passing, as the reference the model is checked against.

That seam is the main design decision, and it would be the same for any model. Decide what the app's commands are, in a form that has nothing to do with language. Then parsing is a replaceable part.

What the grammar cost

The grammar was never trained in the machine-learning sense. It was taught by hand, one sentence at a time, and that's the slow part of the story.

The spaCy pipeline gives each word a part of speech and a dependency link, and everything after that is rules: a vocabulary table of about 290 lines, and an extraction pass of about 690 more that walks the tokens and decides what each one means. The parser's history in the scene engine runs to 17 commits touching the grammar since August, and dowhat's port of it is checked by 54 tests, including a corpus of 66 real phrases. Each phrase that failed exposed a different kind of gap:

  • A parser tagged "floating" as a verb, so a sentence about a floating bear was read as an instruction to float something.
  • A quoted word, as in add the word 'hello', parsed correctly and was then never used, so the scene got an object literally labeled "word".
  • "Move it to the left" had no meaning, because the grammar only knew a row and column together, or the middle.
  • Two grammars, written for different purposes, both claimed "remove." One meant to delete an object and the other a connection between objects.

None of these is exotic, and a person finds each one only by typing the sentence. A rule-based parser has no ceiling on how long that goes on, because speech has no ceiling either. The ratio matters: the model path was one commit, a prompt, a schema and fifteen examples. For a first version the model is much faster to stand up.

The grammar's work still counted, in two ways. The 66 phrases in dowhat's copy (the scene engine's own file has since grown to 86) are a test corpus for any parser that comes next, and the typed command list the grammar was forced to define is the contract the model now fills in.

Faster ways to get there

If writing the rules is the cost, there are four ways around it, in rough order of effort.

  • A prompt and a schema. What dowhat does now. No labeled data, no training, and it works the day it's written. The price is a model on every request and the schema discipline described below.
  • A pretrained model tuned for tool calls. Small models exist whose whole job is to turn a request and a list of tool definitions into a call. Google's FunctionGemma is one, at 270 million parameters, built on Gemma 3. NVIDIA's documentation describes fine-tuning it on a single GPU with the xLAM function-calling dataset, and one write-up puts it near 288 MB against about 1.5 GB for Hammer 1.5B, which it says scores about 73% out of the box. Those figures come from a single blog post and want checking. It sits between the grammar and a general model, smaller and quicker than a 3B chat model.
  • Fine-tune on your own phrases. Where a prompt isn't enough, a tiny model can learn the specific vocabulary from examples. The phrase corpus the grammar produced is the natural training set, since every line is already paired with the commands it should become. This hasn't been tried in dowhat, so it's a plan and not a result.
  • A managed NLU service, or a classic intent-and-slot model. Services such as Amazon Lex take a list of intents and slots with example phrases, and its assisted NLU option uses a language model to sharpen intent and slot matching inside those lists. The open-source equivalent is a pretrained encoder such as BERT with joint intent and slot heads. Both need labeled examples, and the managed ones tie the app to a vendor. For a few dozen commands, and for open vocabulary like "wood duck," they're more machinery than the job needs.

For any of the four, the commands should be written down first as typed data, as above. Swapping the parser is then a one-function change, the same as the swap from spaCy to Ollama.

Words for space and time

A grammar for graphics control spends most of its effort on one problem. Speech is qualitative and the engine is quantitative. Nobody says "put a camel at 83, 83." They say "lower right," "a bit higher," "make it bigger," "faster," "when the star appears." The parser's real job is a mapping from a small vocabulary of loose words to numbers and events the engine can use. The engine's model of an object has many more dials than the command list reaches today, and the dials fall into two families, space and time.

Space

Position. The scene is a 0 to 100 grid with y growing downward, as on a screen, so "lower" means a high y. Regions are a three-by-three table: "upper left" is (17, 17), "middle center" is (50, 50), "lower right" is (83, 83), with bare "left" and "right" meaning vertically centered. Coordinates are clamped to 5 and 95 so nothing spawns off the edge. Asking for several objects in a region scatters them on a jittered ring whose radius grows with the count, because "a swarm of 30" in one point would be one blob.

Relative change. "Raise the bear" and "move the bear to the upper left" are different commands. One is a fixed step from wherever the bear is now and the other is an absolute place, and the schema keeps them apart as nudge and move. Size works the same way. There are three presets, small, medium and large, at 8, 16 and 30 units. "Make it large" picks one directly, and so does "add a large moose," since spawn takes a size word too. "Grow it" finds the preset nearest the object's current size and steps one up, stopping at the ends. Relative words depend on the state of the scene, which is another reason state belongs in code.

Body and layout. An object in the scene spec also carries rotation, mass, shape, stacking order and whether it can be dragged. Of these the command language reaches flip and, through the connection commands, the rope, chain, rod or spring between two objects. Rotation, mass and layering are in the engine and have no words yet. Each would be one more command or one more field, and "rotate it a quarter turn" is the kind of sentence that arrives the first day anyone uses the thing.

Content. Some objects carry text, a speech bubble, a scroll, a pair of words in a dichotomy. This is the one family where the model's free text passes straight through to the scene and doesn't pick from a list. The current schema gives spawn a words array for it, and a force_words flag for sentences like "add words that say chapter 1," where the noun is the text. Most of that prompt exists to stop a small model splitting "to be or not to be" into six words. Text that reaches the screen is untrusted input, so it should be rendered as plain text and never as markup.

Switches and groups. Gravity and repel are booleans, with one subtlety that cost a fix. Spawning something already floating is a different statement from toggling gravity on an object that exists. The first is a property at creation and the second is an event, so the schema has a separate gravity_state field on spawn for the first and a gravity command for the second. Groups are a count plus a layout. Quantity is an integer, and swarm, carousel, row and chain are the four cluster types.

Time

Space is a set of snapshots. Time is where graphics control gets harder, because what's being described continues after the sentence ends. The engine has three kinds of it, and they want three different shapes of command.

Rate. One tick rate is shared by the tick-driven animations in a scene, and the on-screen control for it has three actions: back to normal speed, nudge by a step, jump to the maximum. That is already a command grammar. "Normal speed," "faster," "a little slower" and "as fast as it goes" map straight onto it, as a relative step on a single number, much like grow and shrink. No schema field for it exists yet.

Events. The scene has named signals. Any component can raise one, a string such as a star finishing its growth, and other components wait on it. A morph that cross-fades one drawing into another carries a duration in seconds and can wait for a signal before it starts. The chat grammar already has a trigger command with a signal field. This is time as cause and effect, "when the star appears, turn the bear into the standing bear," and it needs a command that links one event to another, which is a different shape from a number.

Rhythm. The engine also has a scheduler for Strudel mini-notation, the pattern language used in live coding, ported over from the music-composer. A string like bd sn hh sn or 1 0 1 1 0 ~ fires callbacks in sequence at a set number of cycles per second, and a looping component pulses a circle on each beat. Nothing in the first-party scene uses the scheduler yet beyond a demo and that looper. What's interesting is the vocabulary. A pattern is already a small language with its own parser. A command could carry a pattern string as a field, "make the lights pulse like 1 0 1 1," and the pattern parser, not the model, would decide whether it's valid.

That's also the composer's own subject. Its unit of composition is a pattern of notes, and the tempo of the music is the same kind of number as the scene's tick rate. Voice commands for the composer, "a bit slower," "bring the motif back inverted," "from the second chorus," are the same three shapes: a step on a rate, a position in time, and a pattern passed through.

One rule for every family

The model should say the word and the code should own the number. A small model is good at choosing among named options and poor at consistent arithmetic, so the schema asks it for symbols and the translator turns symbols into coordinates, steps and events. The schema can go further than it does today. The region field is a free string that a function parses, and listing the legal regions as an enum would let the decoder reject anything else.

What comes back

The contract with the model is a JSON array of tagged objects, one per command the sentence implies, in order. For the camel sentence it's this:

[
  {"op": "spawn", "noun": "camel", "quantity": 1, "region": "lower right"},
  {"op": "gravity", "value": true},
  {"op": "repel", "value": true}
]

Three mechanisms make that reliable enough to build on.

The schema is enforced while the model writes. Ollama accepts a JSON Schema in its format field and restricts what the model can emit to text that matches it, so the reply can't be malformed JSON, can't be a bare object when an array was asked for, and can't carry an op that isn't on the list. The commands are also described in the system prompt, in prose with a few examples, with the temperature held at 0.1. The schema guarantees the shape and the prompt tells the model what each field means.

Shape isn't meaning. A schema-valid reply can still be wrong. The region field is a free string that a small function maps onto the grid, and noun is open vocabulary on purpose, so the model can say "camel" or "wood duck" and the resolver decides what art that is. Two checks follow. The schema is the first. The second is a validation step in the translator, where each element is checked on its own, and a bad one is skipped with a reason attached. The rest of the array still applies. A single object would have lost the whole sentence over one bad field.

Order carries meaning. "It" in have it connect to gravity means the spawn just before it in the array. Position was chosen over explicit ids because a 3B model is more reliable at putting things in order than at inventing and reusing identifiers. The cost is a dependency that no schema can express, and it lives in the translator.

Is JSON the right mechanism?

For this job, probably so, for a reason that has nothing to do with this model. A JSON Schema with a tagged union is the one description that every rung of the ladder below accepts, though each engine supports a different subset of the schema language and the exotic parts need testing:

  • Ollama takes it as format, as above.
  • Chrome's Prompt API takes it as responseConstraint.
  • WebLLM has a JSON mode with custom schemas.
  • A hosted model takes each branch as a tool definition: op becomes the tool name and the fields become its input schema. Anthropic's, Google's and OpenAI's APIs all work this way, and a reply with several tool calls is the same flat array.

That last point suggests a better design, not yet built. Today the schema and the prose prompt are written by hand, separately, and they can drift. The intent classes are already typed, so a table of command name to schema, generated from them, could produce the Ollama schema, the prompt text and the hosted tool definitions from one source.

The alternatives, and where each fits:

  • Tool calling instead of a schema. The same thing, with better training behind it on hosted models. A small local model's tool support is patchier, which is why dowhat uses the schema. The loop that makes tool calling an agent, where the result goes back to the model and it continues, isn't needed here. Commands are fire and forget, and the scene state stays in code.
  • A line-based mini-language. spawn camel lower-right is fewer tokens than the JSON for the same command, which matters for latency on a local model, since it generates one token at a time. Constrained decoding can enforce a grammar like that too. The price is writing and maintaining a parser, and losing the ecosystem around JSON. With 15 commands JSON is the better fit, though the difference hasn't been benchmarked.
  • Newline-delimited JSON. One command object per line, so the app can apply each as soon as its line completes and the scene starts responding while the model is still writing. For anything with audio, where a half-second matters, that's worth trying.
  • No generation at all. Embed the sentence and pick the nearest known phrase. It's the cheapest rung of all and it can't hallucinate a command, but it has no slots, so "put the cello on the left one" can't carry its argument.

One gap in the current schema: there's no "can't do that" command. An empty array is the only way to decline, and a model forced to answer will find a field to put its guess in, which is how the bear came out standing. A command that means unsupported, with a reason, would give it somewhere honest to go.

What a 3B model gets wrong

The model doesn't need to be clever. It needs to be pinned down, and watching it fail shows three ways.

A loose schema gets empty answers. Asking for "some JSON" gave valid JSON that said nothing, every element just {"op": ..., "value": true} with no noun and no place. Ollama can constrain the output to a JSON Schema, and what worked was a strict one, with a separate branch for each of the fifteen commands, each with its own fields and its own required list. A small model fills in a form well. It does poorly with a blank page.

Meaning goes wherever there's room. The spawn command had no field for "floating." Given "create a floating bear," the model put the meaning in the nearest field it had and returned a bear with the pose "standing." The fix was to add the field. If a word in the sentence matters, the schema needs a slot for it, because the model will find one regardless.

Don't ask it to do the bookkeeping. "Make it float" has a pronoun. The model is never asked which object "it" means. It emits a flat array of commands in order, and plain code tracks the most recent spawn earlier in the array. Counting, referring back and checking ranges are things code does well and a small model does badly, so the model only does the part that's about language.

And the failure mode is boring on purpose. If Ollama is down, or the output is malformed, the result is an empty command list and a note in the reply saying it couldn't understand that as a command. It can't raise an error into the socket, and the scene is untouched.

Hands on the keys

Dowhat has a second input, and it's the more fun one. A piano mode takes MIDI from a real keyboard, using the Web MIDI API, and uses it as the control. It's inspired by PianoText, a research project from Saarland University that repurposes a piano keyboard's chords and note sequences for text entry.

Chords are recognized by shape, so they work from any root, and each shape is a command: a major triad spawns an object, with the root note choosing the region of the screen; an augmented triad spawns a connected pair; minor toggles gravity on the last object; diminished toggles repel. Single notes played in sequence spell the noun, letter by letter. The page assembles an English sentence from what was played and hands it to the same pipeline the text box uses. There's no separate logic for piano.

That's the music-composer's interface turned inside out. Here a pianist's hands control a scene. In the composer, someone with both hands on three keyboards would like to control the music without letting go, by saying "put the cello on the left one and take it from the top." The commands there are record, stop, play, seek, an instrument per keyboard, and the metronome, each already a message on the WebSocket the buttons use. The composer's larger idea is that the unit of composition is a pattern of notes, a leitmotif transformed as it plays, and when that's ready "bring the motif back, inverted" is one more command with a parameter. The list stays short and the same design applies.

The cheap rungs

Running Ollama on a machine already at hand costs nothing but the machine. That's one rung of several, and which one fits depends on who's using the app. The same command schema goes up every one of them, which is the reason to write it as a schema.

  • Self-hosted small model. What dowhat uses. Free of fees, needs a box with the memory for it, and the model never leaves your hardware.
  • The browser's own model. Chrome's Prompt API runs Gemini Nano on the visitor's machine, and per Chrome's documentation it takes a JSON Schema through a responseConstraint option, the same schema trick as above. It's listed at Chrome 148 for the web, wants at least 22 GB of free disk, a GPU with more than 4 GB of VRAM or a 16 GB CPU, and doesn't run on Android or iOS. A bonus for those who have it.
  • A model shipped to the browser. WebLLM runs open models over WebGPU. Models of 0.5 to 1.5 billion parameters come to roughly 280 to 840 MB, by tutorial figures, cached after the first visit. Its README lists JSON-mode generation as supported and function calling as work in progress.
  • A free hosted tier. Gemini Flash, Groq and Cloudflare Workers AI each offer one, with limits that change and sources that disagree. At the time of writing, Groq was about 30 requests a minute and Cloudflare 10,000 "neurons" a day. Put the key behind a route on your own server and count requests per member.
  • Pennies. Anthropic lists Claude Haiku 5.5 at $0.10 per million input tokens and $0.50 per million output tokens for prompts under 100,000 tokens. A command is about 800 tokens in (the schema, the scene state, the sentence) and 50 out, around a hundredth of a cent, so ten thousand commands cost about a dollar. A spending cap makes that safe to leave running.

For a control menu this short, any of these will work, as long as the schema is strict.

The guardrails are the interface

None of the rungs changes the rule that makes this safe. The model's output is a request, and the app decides whether to honor it. A sentence typed into a text box is untrusted input, and what a model makes from it is untrusted too.

  • A closed list of commands. The schema is an enum. Anything outside it is dropped.
  • Validation on the way out. Regions are on the grid, instrument names are in the catalog, seek times are inside the take. The same checks the buttons already apply.
  • The same path as the controls. The result becomes the same patch or message a click would produce, and nothing else is exposed.
  • A confirmation for anything destructive. Replacing a take is one a person should approve.
  • Gating and limits. The composer page is already members-only, and a route can rate-limit per member.

With those in place, the worst a hostile or confused model can do is press something the person could have pressed.

Where to start

Write the commands down as data, with nothing about language in them. Build the dull parser, the grammar or the keyword table, because it gives you the validation, the plumbing and a test corpus. Then put a model behind it on the cheapest rung that works, with a strict schema and no bookkeeping, and let the grammar check it.

Scotty gets the computer to work in the end, with the keyboard. The odd part of the piano mode is that it takes him at his word, and the keyboard is the instrument.

Sources: Star Trek IV: The Voyage Home (Paramount, 1986). Author's projects dowhat, scene-engine and art-synthetic. Saarland University, PianoText. Chrome for Developers, Prompt API and structured output for the Prompt API. NVIDIA, Function Calling with NeMo AutoModel using FunctionGemma. Google Developer Experts, On-device function calling with FunctionGemma. Amazon Web Services, Assisted NLU in Lex V2. WebLLM README. OpenRouter, Free LLM APIs Compared. Anthropic, pricing.

The sentences here were drafted by an AI from the author's direction.