Note: This article is, in several areas, over my head. It has some technical nuances that I don’t have a complete grounding in to articulate it well, but my hope is that it will shed some light on determinism when using AI. @Max_OBrien scratched the itch I had on the topic of determinism several years ago — 2023, I think it was. In the past three years, I’ve had the pleasure of working on AI systems that occasionally cross the Coda path, and in nearly 100% of the work, determinism — specifically involving JSON outputs — is central to my solutions. With that, here are some observations, purely my opinion.
In a recent Coda-related forum, a Maker asked Superhuman AI to output JSON from a price table. Every column, every row. It worked.
Once.
That is the most dangerous kind of success a product can have. It is fluent, it is fast, and it teaches the wrong lesson: that JSON is a tone of voice you request, rather than a contract you enforce.
This is a product argument, not a model argument. You already live in tables. You want an AI column that behaves like a formula: same keys every row, same types, and a miss you can see. Superhuman already has the one thing a chat window does not: a typed table sitting under the cell. Treat JSON as a tool that writes into that table, or “it worked” becomes the feature, and people march on.
Four questions people collapse into one
When someone says “the JSON column works,” they usually mean one of four different things:
| Question | What it actually asks | What a prompt cannot guarantee |
|---|---|---|
| Did it parse? | The string was valid JSON | Extra prose, trailing commas, those code fences |
| Is the schema identical? | Same keys, types, required fields, no extras, every time | The model renaming a column “to be helpful” |
| Are the values true of the row? | Copied fields match the cells they came from | A plausible diesel price that was never in the table |
| Is it byte-identical on rerun? | The same object, twice | Anything an LLM emits |
Christiaan’s diesel table is a projection. The table already has names, types, and cells. Asking a model to retype a table that has already been typed is the wrong job. A formula or a Pack produces 1000/1000 on schema and values. Surprise that the model managed it once is the wrong takeaway.
The same refusal applies to the next job people reach for — “predict tomorrow’s diesel from this series.” That is a spreadsheet job. An LLM is the wrong instrument. Superhuman should make the refusal obvious: if the transform is just math on columns, do not call a model.
JSON Type earns its keep when the job is derivation: classifying a notes column, extracting a messy email into fields, tagging a delay reason. Then a model belongs in the loop. Even then, it does not get to invent the shape.
A type is a stack
“Please reply in JSON” means emit a well-formed object. It does not mean emit this schema. Extra keys, missing required fields, price as a string, a commentary field nobody asked for: all of that still parses.
A type is not a prompt flavor. It is a stack.
Figure 1 — The model proposes a candidate. It never writes. Write-enable lives one layer up in the code, and the maker has to be able to see what happened.
From the bottom:
- Model. Allowed to guess. Forbidden to write the cell.
- Frozen contract. The destination columns, or a schema you declared. Locked before the first row. No extra keys. Required is required. That same contract is sent as a real JSON schema / tool setting on every call, not as a paragraph that says “reply in JSON.”
accept()in code. Parse. Check the contract. Copied fields must match the source cells. This layer is the product. It does not exist if the pipeline stops at “the string looked like JSON.”- Receipts. Attempt number, what failed,
ok. A row you can open the way you open a formula error. - Maker surface. The schema is an editable object. Retries you can set. A hook you can write:
OnJsonFail(receipt).
The layer Superhuman cannot skip is the frozen contract. A prompt never reaches it. An LLM “fixer” that is fed the broken string and asked to try again is not it either. Trailing commas are a library. Missing required keys are retried with the error list. Conflating those two is how a made-up structure gets laundered into a cell.
The constitution of accept()
You cannot make the model deterministic. You can make the gate deterministic:
accept(candidate, frozenSchema, sourceRow) → { ok, value } | { ok: false, errors, receipt }
That function is ordinary code. Retry is another guess against the same schema. Exhaustion is an honest failure. Nothing that fails accept is written as success.
Figure 2 — Two exits Superhuman should treat as first-class: write JSON, or error the cell. There is no third exit called “close enough.”
A few rules that are load-bearing:
Refuse the spreadsheet job. If you just want this row as JSON, do not call a model. The table already has the column names and the values. Export the row from those cells. Then the schema and the numbers match because they came from the table, not from a guess.
Freeze the schema before the first call. Do not infer shape from the first successful response and hope the next 999 match it. Schema identity 1000/1000 is this step, or is it not a type?
Set the JSON switch the way a tool does. A JSON Type should send a real schema to the model, not a please. “Reply in JSON” is a fallback and should be labeled as such.
Parse and validate in the product. Strip fences. Repair syntax with a library. Then check the frozen schema, with path-level errors (price: expected number, got string). Syntax repair is not a model call. Schema repair is a bounded retry, fed those paths, not “here is bad JSON, please fix it.”
Bind values that were copied. Valid JSON can still be a lie. If the field is a column dump — id, date, posted diesel price — the candidate must equal the cell. Invented numbers fail the row. Derived fields (a summary, a tag) cannot be bound this way; they are the only reason the model was invited.
Stop. Default two extra attempts. Then the cell is an error. The raw candidate lives on the receipt. Downstream formulas see a blank, not poison. Infinite hidden repair is how “it worked” becomes an incident.
And if it doesn’t comply?
Then the cell fails. Out loud.
That is the only honest answer, and it is the product surface makers do not have. A JSON Type that retries in the dark trains people to treat a column as reliable because they have not yet seen it miss. Reliability that cannot be inspected is not reliability. It is a streak.
Figure 3 — Written is not the only terminal state. Failed, with a receipt, is the state that keeps the rest of the doc honest.
A cell that cannot produce schema-valid JSON must result in an error with a receipt, just as a formula error does. Not last-best-effort garbage. Not empty-looking success. Not a fourth hidden model call.
Replay has to be deterministic even when generation is not: same row + same schema + same raw candidate → same accept() result. That replay is how a maker debugs, how Superhuman supports them, and how a test suite prevents “it worked a few times” from shipping.
This is also how “we can inspect the returned objects” stops being a sentence only the people who built the router can say. Receipts are how the rest of us participate. Max can already look. Christiaan cannot. The receipts table is the difference.
What Superhuman should ship
Not a research program. A column type that behaves like a tool.
- Compile and freeze a schema from destination column types or from a schema the maker declared. Version it. Stamp every receipt with that version.
- Send a real JSON-schema setting on JSON Type. Prompt-only JSON is a labeled fallback, not the default.
- Validate in the product against that schema, no extra keys. Path-level errors, not “invalid JSON.”
- Retry a small N with the error list. Same frozen schema every attempt. The model does not get to renegotiate shape.
- Error the cell when retries run out. Never write a candidate that failed
accept. A formula downstream is not a substitute for a write gate. - Write a receipt that the maker can open. Attempt, raw (truncated), parse/schema/bind bits, errors, model.
- Let the maker participate. Editable schema. Strict vs coerce. Max retries.
OnJsonFail(receipt)as a formula hook — retry with a different prompt, page someone, write an exceptions table. - Refuse the spreadsheet job in the product. If they want the row as JSON, export the cells. Point numerical forecasts at a real model, not at the LLM.
Until 2, 3, and 5 exist, marketing JSON as reliable is false. A syntax parser plus an LLM rewrite raises parse rates. It does not stabilize the schema. It does not tell the maker what happened. It does not answer “1000 out of 1000, and will the schema be identical?”
Why Superhuman, specifically
Chat side-panels and agent IDEs discover this the hard way, with scripts only a developer will write. Superhuman does not have to.
The table is already typed. Column types already exist. Formula errors already exist. AI columns already run per-row. The missing piece is not another prompt. It is the gate, the frozen schema, and a failure state as visible as #ERROR!.
That is also why a JSON Type is larger than a single column. Workflows that chain AI cells are only as honest as the write gate between them. One drifted object, silently accepted, is a poison pill for every downstream cell. Makers who are not engineers will not write the second line of defense. The product has to be the first.
The labs will continue to improve at structured output. Good. Superhuman should take that as the model layer and still own the contract, the gate, the receipt, and the hook. A provider grammar is not a validator. A validator is not a receipt. A receipt is not a maker hook. Skip any one of those, and JSON is again a suggestion.
The ask
Ship JSON Type as a tool, not as a suggestion.
The model proposes. Code accepts. Rejection is visible. Makers can see the schema, the receipt, and the failure, and they can write a hook. That is the constitution. Everything else — including “it worked” on a pricing table — is a demo.
If Superhuman builds that, 1000/1000 stops being a personality test of the current model and becomes a property of the column. That is the feature. The rest is tone of voice.
As I said, this is just my opinion based on some successes I’ve had building deterministic systems.