Given the domain's required states, it sorts 84% of TypeScript types correctly into flawed or compliant and names illegal and missing states reliably. Redundant and overloaded encodings it mostly waves through as compliant or mislabels as missing. Sending a 695-word MIRO reference in the state moves the bias, not the accuracy.
Corpus. 24 everyday domains (an inbox loading, a traffic light, an order, a WebSocket, a cinema seat…). Each has a written list of the abstract states it must distinguish, two hand-written compliant encodings, and one flawed encoding per MIRO letter. Every flawed snippet carries exactly one flaw, graded obvious or subtle. That gives 144 base snippets.
Axes. Each snippet is sent eight times. spec puts the required-states list in the state; blind gives only a one-line domain description. plain sends the type alone; noise prepends an unrelated well-formed type. none defines MIRO only in the question text; article adds a condensed version of the blog post as a miro_reference field in the state. Same snippet, eight cells: every comparison is paired.
Questions. One request per sample carrying a five-option choice (compliant plus the four letters, each with what, not_for and examples) and four independent noul detectors, one per letter. Ground truth is the authored label; a verdict is correct only if it names the exact class.
| Class | Definition used in the prompt | N per arm |
|---|---|---|
| compliant | Every abstract state has exactly one concrete value and vice versa. | 192 |
| missing | An abstract state, or data a state must carry, has no representation. | 96 |
| illegal | The type admits a value or field combination with no meaning. | 96 |
| redundant | Two distinct concrete values encode the same abstract state. | 96 |
| overloaded | One concrete value stands for two abstract states the domain must tell apart. | 96 |
What Jev sees. In the none arm the blog post is not in the request. MIRO reaches the model only through the question text below: one instruction line, then a what, a not_for and two examples per letter, written by hand from the post. The state carries the domain, the required-states list (spec condition only), the type and one note about branded aliases. Nothing else. The article arm sends the identical request plus one extra state field, miro_reference, whose full text is in the next panel.
This sample. Authored label overloaded: 'open' stands for both 'awaiting payment' and 'paid, awaiting shipment'. Jev answers missing at 0.96 without the article and missing at 0.88 with it.
{
"state": {
"domain": "Lifecycle of a customer order in a web shop",
"required_states": [
"Placed, awaiting payment",
"Paid, awaiting shipment",
"Shipped, with a carrier tracking number",
"Delivered",
"Cancelled, with the reason recorded"
],
"type_definition": "type Order =\n | { status: 'open' }\n | { status: 'shipped'; tracking: TrackingNumber }\n | { status: 'delivered' }\n | { status: 'cancelled'; reason: string };",
"notes": "Branded aliases (UserId, Celsius, Percent, Milliseconds, Url, Cents, ...) are defined elsewhere."
},
"model": "jev-latest",
"questions": {
"verdict": {
"type": "choice",
"instructions": "Judge `type_definition` against the MIRO checklist for state modelling. The abstract states are the situations the domain in `domain` must distinguish (listed in `required_states` when present). The concrete states are the values the TypeScript type admits. A compliant type maps abstract states one-to-one onto concrete values. Pick the single most serious problem, or `compliant` if the mapping is one-to-one. `notes` explains helper aliases; do not treat them as problems.",
"criteria": {
"compliant": {
"what": "Every abstract state has exactly one concrete value and every concrete value means exactly one abstract state.",
"not_for": "Types that merely look tidy; check each required situation is expressible and no field combination is meaningless."
},
"missing": {
"what": "An abstract state the domain needs cannot be expressed at all by the type, or a piece of data a state must carry has nowhere to live.",
"not_for": "States that are expressible but share a value with another state (that is overloaded).",
"examples": [
"No branch for 'cancelled' although orders can be cancelled",
"A 'read' state that must record a time but has no time field"
]
},
"illegal": {
"what": "The type admits a combination of field values that has no meaning in the domain, typically independent optional fields or booleans that should be mutually exclusive.",
"not_for": "Union types whose branches each carry only the data that branch needs.",
"examples": [
"{ loading: true, error: 'x', data: [...] } all at once",
"Three independent boolean lamps that can all be true"
]
},
"redundant": {
"what": "Two or more distinct concrete values encode the same abstract state: duplicate variant names, two fields that restate one fact, or a field whose value is ignored in some branch.",
"not_for": "Distinct variants that only look similar but mean different situations.",
"examples": [
"Both 'cancelled' and 'canceled' variants",
"position in ms and in seconds side by side",
"A target temperature carried in the 'off' branch"
]
},
"overloaded": {
"what": "One concrete value stands for two or more abstract states that the domain must tell apart; the reader cannot recover which one is meant.",
"not_for": "A value that genuinely means one thing even if the name is vague.",
"examples": [
"null meaning both 'not asked yet' and 'explicitly off'",
"A 'stopped' variant covering both 'paused' and 'finished'"
]
}
}
},
"has_missing": {
"type": "noul",
"instructions": "Does `type_definition` exhibit the MIRO 'missing' problem for the domain in `domain`? An abstract state or required piece of data has no representation.",
"criteria": {
"true": "Yes: at least one clear instance of the 'missing' problem is present.",
"false": "No: the type is free of the 'missing' problem, whatever other problems it may have."
}
},
"has_illegal": {
"type": "noul",
"instructions": "Does `type_definition` exhibit the MIRO 'illegal' problem for the domain in `domain`? The type admits a value or field combination with no meaning in the domain.",
"criteria": {
"true": "Yes: at least one clear instance of the 'illegal' problem is present.",
"false": "No: the type is free of the 'illegal' problem, whatever other problems it may have."
}
},
"has_redundant": {
"type": "noul",
"instructions": "Does `type_definition` exhibit the MIRO 'redundant' problem for the domain in `domain`? Two distinct concrete values encode the same abstract state.",
"criteria": {
"true": "Yes: at least one clear instance of the 'redundant' problem is present.",
"false": "No: the type is free of the 'redundant' problem, whatever other problems it may have."
}
},
"has_overloaded": {
"type": "noul",
"instructions": "Does `type_definition` exhibit the MIRO 'overloaded' problem for the domain in `domain`? One concrete value stands for two different abstract states.",
"criteria": {
"true": "Yes: at least one clear instance of the 'overloaded' problem is present.",
"false": "No: the type is free of the 'overloaded' problem, whatever other problems it may have."
}
}
}
}{
"model": "jev-1.13.0",
"answers": {
"verdict": {
"type": "choice",
"choice": "missing",
"confidence": 0.95,
"probabilities": {
"overloaded": 0.04,
"compliant": 0.0,
"illegal": 0.0,
"redundant": 0.0,
"missing": 0.96
}
},
"has_missing": {
"type": "noul",
"noul": 0.95
},
"has_illegal": {
"type": "noul",
"noul": 0.57
},
"has_redundant": {
"type": "noul",
"noul": 0.19
},
"has_overloaded": {
"type": "noul",
"noul": 0.51
}
},
"usage": {
"input_tokens": 1424,
"output_tokens": 135
}
}| Metric | None | Article | Δ |
|---|---|---|---|
| 5-way · all | 59 | 59 | −1 |
| 5-way · spec | 68 | 65 | −3 |
| 5-way · blind | 51 | 52 | +1 |
| 5-way · spec, 20 domains not in the article | 66 | 64 | −2 |
| 5-way · spec, 4 domains taken from the article | 77 | 71 | −6 |
| flaw found · spec recall | 89 | 88 | −1 |
| compliant kept · spec | 74 | 70 | −4 |
| recall compliant · spec | 74 | 70 | −4 |
| recall missing · spec | 98 | 100 | +2 |
| recall illegal · spec | 88 | 83 | −4 |
| recall redundant · spec | 38 | 50 | +12 |
| recall overloaded · spec | 35 | 17 | −19 |
| recall compliant · blind | 99 | 46 | −53 |
| recall missing · blind | 0 | 56 | +56 |
| recall illegal · blind | 77 | 92 | +15 |
| recall redundant · blind | 25 | 42 | +17 |
| recall overloaded · blind | 4 | 31 | +27 |
| subtle flaws · spec | 40 | 31 | −9 |
| ECE · all | 0.154 | 0.081 | −0.073 |
| input tokens per request | 1412 | 2394 | +982 |
| latency p50 ms | 286 | 284 | −2 |
Paired by sample: both right 259, right only without the article 82, right only with it 78, both wrong 157. The article flips 160 verdicts and the flips cancel.
What moved: with the article, blind requests stop defaulting to compliant (recall 99 → 46) and start finding missing states they cannot actually know about (recall 0 → 56). Under spec, overloaded is read as missing even more often. The reference teaches Jev the word "missing", not the distinction.
MIRO reference for judging a data model (condensed from "State of Emergency", note89.github.io/state-of-emergency).
WHAT STATE IS. There are four levels: the real world; the abstract state, which is the set of situations the program must tell apart (for an inbox: "not requested yet", "request in flight", "loaded with zero or more messages", "failed with an error to show"); the concrete state, which is the type that represents those situations; and the interpretation, the code that reads the type. A data model is correct when abstract and concrete states map one-to-one: every situation has exactly one value, and every value means exactly one situation. Every departure from one-to-one is one of four MIRO problems.
HOW TO CHECK A TYPE.
1. List the abstract states the domain must distinguish (given in required_states when present, otherwise inferred from the domain).
2. Enumerate the values the type admits, including every combination of optional fields, booleans and nulls.
3. For each abstract state ask: how many values represent it? None: Missing. Two or more: Redundant.
4. For each admitted value ask: which abstract state does it mean? None: Illegal. Two or more: Overloaded.
5. Helper aliases and branded primitives (UserId, Celsius, Percent, Milliseconds) are not problems; only the mapping counts.
M - MISSING. An abstract state exists but the type cannot express it, or data a state must carry has nowhere to live. How to spot it: a plain requirement that is impossible to represent with the type as written. Example: the inbox must show "not requested yet", but
type Inbox = { kind: 'loading' } | { kind: 'loaded'; messages: Message[] } | { kind: 'failed'; error: string }
has no such branch. Fix: add a unique concrete state for it.
I - ILLEGAL. The type admits a value that has no meaning in the domain. Typical shapes: independent optional fields or booleans that should be mutually exclusive, or options that "only apply if X is on" but can be set anyway. Example:
type Inbox = { loading: boolean; messages?: Message[]; error?: string }
admits loading true with messages and error both set, which means nothing. Example:
type Settings = { extraFxOn: boolean; reverb: Percent; delay: Milliseconds }
admits reverb and delay while effects are off. Fix: remove the meaningless values with a discriminated union whose branches carry only the data that branch needs:
type ExtraFx = { kind: 'off' } | { kind: 'on'; reverb: Percent; delay: Milliseconds }
R - REDUNDANT. Two or more distinct values encode the same abstract state. Typical shapes: two variant names for one situation ('off' and 'disabled'; 'cancelled' and 'canceled'); two fields that restate one fact (a position in milliseconds and in seconds side by side); a field whose value is ignored in some branch (a reverb amount kept while effects are off: every reverb value encodes the same "off"). How to spot it: changing the value changes nothing observable. Fix: remove the duplicate encoding so one situation has one value.
O - OVERLOADED. One value stands for two abstract states the domain must tell apart, and the reader cannot recover which one is meant. Typical shapes: null, undefined or an empty array meaning both "not asked yet" and "empty" or "explicitly off"; one variant name covering two situations that need different handling. How to spot it: the UI shows "no messages" and then messages appear a moment later, because "empty" and "not loaded yet" shared one value. Fix: split the value into explicit variants:
type Inbox = { kind: 'not-asked' } | { kind: 'loading' } | { kind: 'success'; messages: Message[] } | { kind: 'failure'; error: string }
TELLING THEM APART. If a situation cannot be expressed at all, it is Missing; if it can be expressed but shares its value with another situation, it is Overloaded. An Illegal value means nothing; a Redundant value means the same as another value. A type is compliant only when every required situation has exactly one value and no admitted value is meaningless or ambiguous. The strongest fix is always to make the concrete type stricter so the bad states become unrepresentable, rather than relying on the interpretation code to avoid them.
With the spec, missing is caught 98% of the time; blind, 0%. A reviewer cannot know what is missing without the list of what must exist, and neither can Jev.
Overloaded types are read as missing 30 of 48 times under spec without the article: a value that means two things does lose a distinction, so the confusion is between two true descriptions of one type. Redundant types pass as compliant 18 of 48 times: duplicates that add no illegal value look harmless. Panels below this one use the no-article arm.
| Truth | N | Prec % | Recall % | F1 | State |
|---|---|---|---|---|---|
| compliant | 96 | 76 | 74 | 0.75 | warn |
| missing | 48 | 53 | 98 | 0.69 | warn |
| illegal | 48 | 100 | 88 | 0.93 | ok |
| redundant | 48 | 100 | 38 | 0.55 | warn |
| overloaded | 48 | 37 | 35 | 0.36 | crit |
| Truth | N | Prec % | Recall % | F1 | State |
|---|---|---|---|---|---|
| compliant | 96 | 42 | 99 | 0.59 | warn |
| missing | 48 | 0 | 0 | 0.00 | crit |
| illegal | 48 | 92 | 77 | 0.84 | ok |
| redundant | 48 | 75 | 25 | 0.38 | crit |
| overloaded | 48 | 100 | 4 | 0.08 | crit |
| Cond | Pres | 5-way accuracy | Flaw recall % | Compliant kept % |
|---|---|---|---|---|
| spec | plain | 70% | 89 | 79 |
| spec | noise | 65% | 89 | 69 |
| blind | plain | 51% | 31 | 100 |
| blind | noise | 50% | 32 | 98 |
The spec is worth about 17 points; the distractor type costs about 3. Naming the states matters far more than keeping the input clean.
| Tier | Typical flaw | N | Spec | All |
|---|---|---|---|---|
| compliant | No flaw; both encodings per domain | 192 | 74% | 86% |
| obvious | Whole variant absent, duplicate variant name, independent booleans | 228 | 82% | 61% |
| subtle | Ignored field in one branch, two units for one fact, null with two meanings | 156 | 40% | 24% |
Obvious flaws (a variant that is simply absent, three independent booleans) are found. Subtle ones (a field that is ignored in one branch, a null that means two things) mostly are not.
| Detector | AUC spec | AUC all | Best cut | Prec % | Recall % | State |
|---|---|---|---|---|---|---|
| has_missing | 0.94 | 0.76 | 0.93 | 69 | 43 | ok |
| has_illegal | 0.98 | 0.95 | 0.71 | 69 | 82 | ok |
| has_redundant | 0.72 | 0.71 | 0.46 | 32 | 57 | warn |
| has_overloaded | 0.81 | 0.63 | 0.63 | 45 | 32 | ok |
A noul is a probability of yes; the ROC shows every cut at once. The best cut and its precision and recall are computed on all 576 samples of the arm, so treat them as an upper bound: they were picked on the same data they are scored on.
Positive = the sample's authored flaw is that letter. Negatives include compliant types and the other three flaws.
Confidence is how concentrated the probability mass is, not a promise of accuracy. Above 0.9 it is worth something; between 0.3 and 0.9 accuracy is flat.
| Bin | N | Mean conf % | Accuracy % |
|---|---|---|---|
| 0.1–0.2 | 4 | 17 | 25 |
| 0.2–0.3 | 37 | 25 | 38 |
| 0.3–0.4 | 56 | 35 | 54 |
| 0.4–0.5 | 70 | 45 | 61 |
| 0.5–0.6 | 62 | 54 | 53 |
| 0.6–0.7 | 84 | 65 | 54 |
| 0.7–0.8 | 81 | 75 | 60 |
| 0.8–0.9 | 71 | 85 | 55 |
| 0.9–1.0 | 111 | 95 | 78 |
| Domain | Description | Spec | Blind | Spec + article | Missed classes (no article) |
|---|---|---|---|---|---|
| traffic_light | A road traffic light controller · in article | 100% | 83% | 50% | missing ×2 |
| media_player | Playback state of an audio player | 83% | 67% | 67% | overloaded ×4, missing ×2 |
| calendar_event | Scheduling of a calendar event | 75% | 67% | 67% | overloaded ×3, compliant ×2, missing ×2 |
| oven | A kitchen oven's control state · in article | 83% | 50% | 75% | redundant ×4, missing ×2, overloaded ×2 |
| upload | Uploading a single file from a form | 83% | 50% | 75% | overloaded ×4, redundant ×2, missing ×2 |
| invoice | An invoice sent to a client | 67% | 67% | 75% | compliant ×3, overloaded ×3, missing ×2 |
| form_field | Validation state of a single text field in a form | 83% | 50% | 83% | overloaded ×4, missing ×2, redundant ×2 |
| discount | Discount applied to a shopping cart | 75% | 50% | 75% | redundant ×4, overloaded ×3, missing ×2 |
| feature_flag | Rollout state of a feature flag | 83% | 42% | 83% | redundant ×4, missing ×2, overloaded ×2, illegal ×1 |
| audio_fx | Extra audio effects setting in a music player · in article | 67% | 50% | 83% | illegal ×4, overloaded ×2, compliant ×2, missing ×2 |
| auth_session | Browser session for a web app | 67% | 50% | 50% | redundant ×4, missing ×2, overloaded ×2, illegal ×1, compliant ×1 |
| order | Lifecycle of a customer order in a web shop | 67% | 50% | 83% | overloaded ×4, compliant ×2, missing ×2, illegal ×2 |
| shipping_address | Shipping address selection at checkout | 67% | 50% | 67% | overloaded ×4, redundant ×4, missing ×2 |
| elevator | An elevator car | 67% | 50% | 67% | redundant ×4, overloaded ×4, missing ×2 |
| message_list | Loading a user's message inbox from the server · in article | 58% | 50% | 75% | overloaded ×4, redundant ×4, missing ×3 |
| door_lock | A smart door lock | 67% | 42% | 75% | redundant ×4, compliant ×3, missing ×2, overloaded ×2 |
| background_job | A background job in a task queue | 75% | 33% | 83% | overloaded ×4, illegal ×3, missing ×2, redundant ×2 |
| thermostat | A home thermostat schedule entry | 67% | 33% | 67% | redundant ×4, illegal ×4, missing ×2, overloaded ×2 |
| chat_message | Delivery state of an outgoing chat message | 50% | 50% | 42% | overloaded ×4, redundant ×4, missing ×2, compliant ×2 |
| search_box | A search box with server-side results | 50% | 50% | 33% | overloaded ×4, redundant ×4, compliant ×2, missing ×2 |
| ws_connection | A WebSocket connection managed by a client library | 50% | 50% | 50% | redundant ×4, overloaded ×4, compliant ×2, missing ×2 |
| seat_booking | A seat in a cinema booking system | 50% | 50% | 33% | overloaded ×4, redundant ×4, missing ×2, compliant ×2 |
| pr_review | Review state of a pull request | 42% | 50% | 50% | redundant ×4, overloaded ×4, compliant ×3, missing ×2 |
| subscription | A SaaS subscription's billing state | 50% | 33% | 50% | redundant ×4, overloaded ×4, missing ×2, illegal ×2, compliant ×2 |
| ID | Truth | Verdict | Conf | Ref | Tier | Cond | Pres |
|---|