SOFTWARE DESIGN · JEV
EVALUATION · TYPESAFE JEV · THE MIRDIN SOFTWARE DESIGN QUIZ

On somebody else's quiz Jev scores 20%, below the 25% you get by guessing. Telling it to answer as a grug-brained programmer moves it to 33% — by making it less sure, not more right.

The three tabs to the left are benchmarks written for this model, and it scores 59%, 88% and 98% on them. This tab is the Mirdin software design quiz, written to catch working engineers. Each of the five questions was sent 40 times with the correct answer rotated through all four letters, then the whole corpus was sent again with one thing changed in the instructions: who the model is told it is. 3 of the 5 questions came back with the identical answer both times.

jev-1.13.0 · 400 REQUESTS IN 2 PROMPT ARMS · 5 QUESTIONS × 4 ROTATIONS × 10 REPEATS · 1 CHOICE PER REQUEST · 2026-09-18
PLAIN PROMPT
20%
CHANCE 25% · 200 REQUESTS
GRUG PROMPT
33%
SAME REQUESTS · +13 POINTS
QUESTIONS THAT MOVED
2 OF 5
3 ANSWERED IDENTICALLY
SAME CHOICE EVERY RUN
20/20
GRUG 16 OF 20
MEAN CONFIDENCE
0.72
GRUG 0.65 · LOWER
COST
0.0123USD
BOTH ARMS · 400 REQUESTS
WHAT THE PROMPT CHANGEDSAME CORPUS · SAME MODEL · ONLY THE INSTRUCTIONS DIFFER

The experiment. The corpus, the four answer texts, the rotations and the model are byte-identical across both arms. The only thing that changes is the instructions string on the choice question — from a neutral "you are taking a software design quiz" to the grug-brain voice on the right, which names complexity as the enemy and premature abstraction as its disguise. No hints about any specific question, no rubric, no reference text.

What moved. Accuracy went 20% → 33%: 30 answers gained on q2_rectangle, 4 given back on q4_post_comments, net 26 of 200. On q2 the grug prompt drops the IRectangle interface answer and picks making illegal states unrepresentable 75% of the time. On the three it already got wrong — q1, q3, q5 — it picked the same wrong option in all 40 grug runs as in all 40 plain ones. The persona did not teach it anything about those.

What it cost. The plain arm is perfectly stable: 20 of 20 groups return the same choice in all 10 runs, and the letters come out 50/50/50/50 — flat, no position effect at all. The grug arm is stable in only 16 of 20, its letters skew to 60/54/43/43, and mean confidence falls from 0.72 to 0.65. On q4, the one question the plain prompt got right every time, the grug prompt now gets it wrong 10% of the time.

How to read that. The persona loosened the model's grip rather than improving its taste. Where its confident answer was wrong, the wobble had somewhere better to land and accuracy rose; where its confident answer was right, the same wobble cost it. Note q1, where grug is more confident than plain (0.61 → 0.79) and still wrong every run. A prompt that buys 13 points by trading determinism for noise is not a prompt you would ship.

PLAIN ARM · INSTRUCTIONS20%
You are taking a software design quiz. `context` introduces the situation, `code` is the snippet under discussion and `question` is what is being asked. Each criterion below is one of the four offered answers, quoted verbatim. Pick the single best answer. Judge the answers on software design merit, not on how they are worded or ordered.
GRUG ARM · INSTRUCTIONS33%
you are grug brain developer. grug not smart like 10x complexity wizard, but grug ship many project and grug remember what hurt later. grug greatest enemy is complexity demon spirit: complexity demon enter codebase through small crack, grow quiet, then one day grug cannot change one line without eight other thing break. grug also know premature abstraction is complexity demon wearing nice coat, and grug know sometimes boring obvious thing is correct thing.
`context` say the situation. `code` is the snippet. `question` say what is asked. each criterion below is one of four answer offered, word for word. grug pick single best answer. grug judge answer on what happen to this code later, not on how answer worded and not on which letter it sit at.
QuestionPrincipleTruthPlain pickPlain accuracyConfGrug pickGrug accuracyConfState
q1_sanitizeimplementation/interface distinctionanswer BC
0%
0.61C
0%
0.79IDENTICAL
q2_rectanglePlain English Test / Representable-Valid Principleanswer AC
0%
0.88A
75%
0.49MOVED
q3_display_typealgebraic refactoring / a data structure wants outanswer BC
0%
0.84C
0%
0.82IDENTICAL
q4_post_commentsParnas Subset Criteriaanswer BB
100%
0.64B
90%
0.53MOVED
q5_xor_digeststhe design of software is a thing apartanswer DA
0%
0.61A
0%
0.63IDENTICAL

Picks are un-rotated back to the original quiz letters. Accuracy is over all 40 requests for that question in that arm.

DOES IT GIVE THE SAME CONFIDENCE EVERY TIME?THE DIRECT ANSWER · 20 GROUPS × 10 IDENTICAL REQUESTS

No — but the answer never changes. The same request was sent 10 times in a row, 20 times over. In 20 of 20 groups the model picked the same letter in all 10 runs. In 0 of 20 groups the four probabilities came back identical. So the choice is stable and the numbers behind it wobble, by at most 0.034 standard deviation within a group. Ranking never flips; the third decimal is noise.

Where confidence really moves is the letter. On q4_post_comments — the only question it gets right — the probability on the correct answer runs from 0.46 to 0.80 depending on which of A/B/C/D that answer is printed on, a swing of 0.35. The pick is unaffected. Read the confidence as a rank within one request, never as a number you can compare across requests or threshold on.

Practical consequence: repeats buy nothing. Majority vote over 10 runs scores 20%, exactly the same as a single run. That is why every other tab on this site sends each case once.

Questionp(answer) · rot 0rot 1rot 2rot 3SwingWorst run-to-run σ
q1_sanitize0.20B0.27C0.23D0.23A0.06±0.032
q2_rectangle0.15A0.06B0.05C0.04D0.11±0.020
q3_display_type0.06B0.03C0.01D0.04A0.04±0.014
q4_post_comments0.80B0.58C0.46D0.73A0.35±0.034
q5_xor_digests0.03D0.12A0.05B0.03C0.09±0.011

Small letter under each number is where the correct answer was printed in that rotation. The swing column is the spread across letters; the last column is the spread across 10 identical requests.

WHAT WAS TESTEDTHE QUIZ · THE ROTATION DESIGN

The quiz. Five TypeScript multiple-choice questions from the Mirdin software design self-assessment, transcribed verbatim — context, code, question and all four options. Nothing was reworded. The four options for each question become the four criteria of a single choice question, which is the same primitive the other three tabs use.

The rotation design. A model that likes the letter C scores 25% on a quiz where the answer is always C, without knowing anything. So each question is sent in four rotations: the correct answer sits on A, then B, then C, then D, with the other three options moved in lockstep. Each rotation is repeated 10 times with a byte-identical request, giving 200 requests and 20 determinism groups.

The result is not a position effect. Across 200 requests the letters were chosen 50/50/50/50 — perfectly flat — and accuracy is 20% at every rotation. The model tracks the content of each option, not its position. It simply prefers the wrong content, and prefers it consistently. That flatness is a property of the plain prompt: under the grug prompt the same corpus comes back 60/54/43/43, so a prompt that loosens the model's certainty lets position start to matter.

RotationNAccuracyTimes that letter chosen
rotation 050
20%
50
rotation 150
20%
50
rotation 250
20%
50
rotation 350
20%
50
Confidence binNMean confAccuracy %State
0.4–0.5130.4669warn
0.5–0.6390.5521crit
0.6–0.7380.648crit
0.7–1.01100.8318crit

The calibration table is the uncomfortable one: the 110 most confident answers (mean 0.83) are right 18% of the time, while the least confident bin is right 69%. On this quiz confidence points the wrong way — high confidence marks the questions where the plausible-sounding distractor wins.

ONE REQUEST, VERBATIMq4_post_comments.rot0.r00 · 249 MS · 662 TOK IN

The one it gets right. The options are sent verbatim as the four criteria — no hints, no rubric, no reference text. Jev answers B at 0.78. The same dependency, asked as an open five-way classification instead of a multiple choice, is on the Parnas tab.

REQUEST · POST api.typesafe.ai/v1/systemoneAUTHORIZATION: BEARER …
{
  "state": {
    "context": "Consider the following code that you might find in a blog or news website.",
    "code": "class Post {\n  private comments: Comment[];\n  constructor(\n    private id: number,\n    private content: string,\n    private author: string,\n    comments: Comment[]\n  ) {\n    this.comments = [...comments]; // Create a shallow copy of the comments array\n  }\n  getId(): number { return this.id; }\n  getContent(): string { return this.content; }\n  setContent(content: string): void { this.content = content; }\n  getAuthor(): string { return this.author; }\n  getComments(): Comment[] {\n    return [...this.comments]; // Return a shallow copy of the comments array\n  }\n  addComment(c: Comment): void {\n    this.comments.push(c);\n  }\n}",
    "question": "Which of the following proposed changes is most likely to improve the maintainability of this code and the rest of the system? Note that this code may contain other flaws not listed among these choices."
  },
  "model": "jev-latest",
  "questions": {
    "answer": {
      "type": "choice",
      "instructions": "You are taking a software design quiz. `context` introduces the situation, `code` is the snippet under discussion and `question` is what is being asked. Each criterion below is one of the four offered answers, quoted verbatim. Pick the single best answer. Judge the answers on software design merit, not on how they are worded or ordered.",
      "criteria": {
        "A": "The id of a Post should be a hash of its content and author",
        "B": "The Post class should not contain a list of Comments",
        "C": "The comments list should not be copied",
        "D": "The code should not assume that all Posts have authors"
      }
    }
  }
}
RESPONSE · HTTP 200jev-1.13.0
{
  "model": "jev-1.13.0",
  "answers": {
    "answer": {
      "type": "choice",
      "choice": "B",
      "confidence": 0.7,
      "probabilities": {
        "D": 0.14,
        "B": 0.78,
        "A": 0.01,
        "C": 0.07
      }
    }
  },
  "usage": {
    "input_tokens": 662,
    "output_tokens": 45
  }
}
PER QUESTION40 REQUESTS EACH · LETTERS SHOWN ARE THE ORIGINAL QUIZ LETTERS
QuestionPrincipleTruthJevAccuracyp(answer)σConfidenceState
q1_sanitizeimplementation/interface distinctionanswer Bpicked C
0%
0.23±0.030.61wrong
q2_rectanglePlain English Test / Representable-Valid Principleanswer Apicked C
0%
0.08±0.040.88wrong
q3_display_typealgebraic refactoring / a data structure wants outanswer Bpicked C
0%
0.04±0.020.84wrong
q4_post_commentsParnas Subset Criteriaanswer Bpicked B
100%
0.64±0.140.64right
q5_xor_digeststhe design of software is a thing apartanswer Dpicked A
0%
0.06±0.040.61wrong

"Jev" is the option it picked, un-rotated back to the original quiz letter. In every one of the 200 requests, the pick was the same original option — the rotation moved the letter, not the answer.

The four wrong answers have a shape. On q1 it prefers "Option 2 because it's more clear" over the correctness argument; on q2 an IRectangle interface over making illegal states unrepresentable; on q3 named constants over the algebraic refactor; on q5 the reusable-helper answer over "the design of software is a thing apart". Each losing pick is the conventional-wisdom answer — the one that sounds like good practice and that the quiz was built to catch.

q1_sanitize IMPLEMENTATION/INTERFACE DISTINCTION · wrong · 0 of 40

As part of a large application, you are reading an input and need to sanitize it by escaping all single quotes. Which of these two options is better and why? Assume both do the exact same thing.

Option 1: x = sanitize(readInput());
Option 2: x = readInput().replaceAll("'", "\\'");

Multiple benefits of your preferred option may apply. Choose the best answer.

A
Option 1 because it's shorter
0.12
B
Option 1 because option 2 is just plain incorrect, even though it always works
0.23ANSWER
C
Option 2 because it's more clear what it does
0.61JEV
D
Option 2 because it's more efficient, avoiding the function call overhead
0.03

WHY B IS THE ANSWER
The idea of a sanitized string and the idea of a string with escaped quotes are different ideas that happen to share an implementation. Option 2 can only be known to work by relying on a secret of the sanitize module, so it is incorrect as written.

q2_rectangle PLAIN ENGLISH TEST / REPRESENTABLE-VALID PRINCIPLE · wrong · 0 of 40

Consider this Rectangle type as you might find in a GUI framework.

class Rectangle {
  constructor(
    private x: number,
    private y: number,
    private width: number,
    private height: number
  ) {}
  getArea(): number {
    return this.width * this.height;
  }
  getPerimeter(): number {
    return 2 * (this.width + this.height);
  }
}

Which of the following is mostly likely to be a design improvement to this class?

A
The constructor should take two parameters, not four
0.08ANSWER
B
The application should pass around arrays of integers instead of the Rectangle type to avoid the overhead of classes
0.00
C
Rectangle should be made to be an implementation of an IRectangle interface
0.88JEV
D
The class should cache the area and perimeter instead of computing them anew each time
0.04

WHY A IS THE ANSWER
Asked in plain English what data defines a rectangle, people say a corner and a dimension, not four integers. Taking a Point and a Dimension matches that and turns 23 of the 24 argument orderings into type errors.

q3_display_type ALGEBRAIC REFACTORING / A DATA STRUCTURE WANTS OUT · wrong · 0 of 40

Here is a snippet of code you might find in an ecommerce website. Think about what you would do to generally improve this code without changing the behavior.

public setProductDisplayType(displayType: string): void {
    if (displayType === "compact") {
        this.maxDescriptionLength = 140;
        this.numSimilarProductsToShow = 2;
        this.numReviewsToShow = 1;
    } else if (displayType === "detailed") {
        this.maxDescriptionLength = 500;
        this.numSimilarProductsToShow = 5;
        this.numReviewsToShow = 7;
    }
}

Try to think of the best refactoring yourself before looking at the answers. We are deliberately not showing you full detail in the answers below, for fear that showing you the possible solutions would make this question too easy. But if you can think up the best refactoring, then it should be clear which of the answers below is the best match.

A
The best refactored version does the equivalent of looking up a COMPACT_PRODUCT_WIDTH in a hardcoded global hash table
0.12
B
The best refactored version contains some form of the triple (140, 2, 1)
0.04ANSWER
C
The best refactored version replaces the hardcoded strings with constants COMPACT and DETAILED, defined either as String constants or as an enum
0.84JEV
D
The best refactored version uses a switch statement instead of an if-statement
0.00

WHY B IS THE ANSWER
The three numbers travel together, so they are one configuration object. Passing that object is the only answer that truly removes the conditional instead of renaming it.

q4_post_comments PARNAS SUBSET CRITERIA · right 100%

Consider the following code that you might find in a blog or news website.

class Post {
  private comments: Comment[];
  constructor(
    private id: number,
    private content: string,
    private author: string,
    comments: Comment[]
  ) {
    this.comments = [...comments]; // Create a shallow copy of the comments array
  }
  getId(): number { return this.id; }
  getContent(): string { return this.content; }
  setContent(content: string): void { this.content = content; }
  getAuthor(): string { return this.author; }
  getComments(): Comment[] {
    return [...this.comments]; // Return a shallow copy of the comments array
  }
  addComment(c: Comment): void {
    this.comments.push(c);
  }
}

Which of the following proposed changes is most likely to improve the maintainability of this code and the rest of the system? Note that this code may contain other flaws not listed among these choices.

A
The id of a Post should be a hash of its content and author
0.01
B
The Post class should not contain a list of Comments
0.64ANSWER · JEV
C
The comments list should not be copied
0.10
D
The code should not assume that all Posts have authors
0.25

WHY B IS THE ANSWER
A useful subset of the program has posts but no comments, so the Post class must not depend on the Comment class. Copying the list is correct, not a flaw.

q5_xor_digests THE DESIGN OF SOFTWARE IS A THING APART · wrong · 0 of 40

Following is code you might find in a distributed system. A digest in this context is a hash of information relating to the cluster. This function is used to simultaneously compare an unordered set of objects (files, messages, etc) against a checksum. It first combines the digests of each object by taking their XOR, and then compares the result to the target checksum, also using XOR. Note that, while there are many disadvantages to using XOR as a digest, this example is adapted from a real codebase where those were deemed not important.

function compareDigestsToTarget(digests: number[], target: number): boolean {
  let combinedDigest = 0;
  for (const curDigest of digests) {
    combinedDigest = combinedDigest ^ curDigest;
  }
  const difference = combinedDigest ^ target;
  return difference === 0;
}

Choose the best of the following proposed code improvements. There may be other useful improvements not named.

A
The target parameter should be merged into the digests[], so that line 8 can be removed
0.61JEV
B
The code should be optimized to have one less loop iteration, initializing combinedDigest to digest[0]
0.10
C
The two uses of the XOR (^) operator should be extracted into a function
0.23
D
The two uses of the XOR (^) operator should be extracted into two different functions
0.06ANSWER

WHY D IS THE ANSWER
The two XORs have the same implementation but different specifications - combining digests and comparing digests. Identical code with different specs must not be merged.

EVERY RUNCLICK A ROW FOR THE FULL DISTRIBUTION OF THAT RUN
IDQuestionShown answerChosep(answer)p(chosen)Rot
WHAT THIS MEANS FOR THE OTHER THREE TABSREAD THIS BEFORE TRUSTING A BENCHMARK NUMBER

The authored benchmarks name their classes and describe them in the question criteria. "Overloaded means one representation carries two meanings, for example …" is most of the answer. This quiz names nothing: it shows two snippets and four opinions, and the wrong opinions are the ones a competent engineer would defend in review. That gap — 88% and 97% on the authored tests, 20% here — is the honest headline of this whole site.

What survives the gap is narrow and real: given an explicit taxonomy and one snippet, Jev sorts code into that taxonomy quickly, cheaply and repeatably, and it does not cry wolf on clean code. What does not survive is unprompted design judgment. Do not put it in front of a diff and ask "is this good?".

One more caution about this tab specifically: five questions is a tiny sample, and a published quiz may well have appeared in training data — which would make 20% a floor, not a ceiling, and makes the 20% harder to explain away, not easier.