SOFTWARE DESIGN · JEV
EVALUATION · TYPESAFE JEV · THE MIRDIN SOFTWARE DESIGN QUIZ

On a real quiz Jev scores 20% — below the 25% you get by guessing — and the one question it gets right is the Parnas one

The three tabs to the left are benchmarks written for this model, and it scores 59%, 88% and 98% on them. This tab is somebody else's quiz, written to catch working engineers. Each of the five questions was sent 40 times, with the correct answer rotated through all four letters. 1 of 5 came back right, the same 1 every single time.

jev-1.13.0 · 200 REQUESTS · 5 QUESTIONS × 4 ROTATIONS × 10 REPEATS · 1 CHOICE PER REQUEST · 2026-09-17
ACCURACY
20.0%
CHANCE 25 · N 200
QUESTIONS ANSWERED RIGHT
1 OF 5
ONLY THE PARNAS ONE
SAME CHOICE EVERY RUN
20/20
10 IDENTICAL REQUESTS PER GROUP
IDENTICAL CONFIDENCE
0/20
MAX STDEV 0.034
MAJORITY VOTE
20.0%
NO GAIN FROM REPEATS
COST
0.0058USD
128K TOK · 641 TOK/REQ
DOES IT GIVE THE SAME CONFIDENCE EVERY TIME?THE DIRECT ANSWER · 20 GROUPS × 10 IDENTICAL REQUESTS

No — but the answer never changes. The same request was sent 10 times in a row, 20 times over. In 20 of 20 groups the model picked the same letter in all 10 runs. In 0 of 20 groups the four probabilities came back identical. So the choice is stable and the numbers behind it wobble, by at most 0.034 standard deviation within a group. Ranking never flips; the third decimal is noise.

Where confidence really moves is the letter. On q4_post_comments — the only question it gets right — the probability on the correct answer runs from 0.46 to 0.80 depending on which of A/B/C/D that answer is printed on, a swing of 0.35. The pick is unaffected. Read the confidence as a rank within one request, never as a number you can compare across requests or threshold on.

Practical consequence: repeats buy nothing. Majority vote over 10 runs scores 20%, exactly the same as a single run. That is why every other tab on this site sends each case once.

Questionp(answer) · rot 0rot 1rot 2rot 3SwingWorst run-to-run σ
q1_sanitize0.20B0.27C0.23D0.23A0.06±0.032
q2_rectangle0.15A0.06B0.05C0.04D0.11±0.020
q3_display_type0.06B0.03C0.01D0.04A0.04±0.014
q4_post_comments0.80B0.58C0.46D0.73A0.35±0.034
q5_xor_digests0.03D0.12A0.05B0.03C0.09±0.011

Small letter under each number is where the correct answer was printed in that rotation. The swing column is the spread across letters; the last column is the spread across 10 identical requests.

WHAT WAS TESTEDTHE QUIZ · THE ROTATION DESIGN

The quiz. Five TypeScript multiple-choice questions from the Mirdin software design self-assessment, transcribed verbatim — context, code, question and all four options. Nothing was reworded. The four options for each question become the four criteria of a single choice question, which is the same primitive the other three tabs use.

The rotation design. A model that likes the letter C scores 25% on a quiz where the answer is always C, without knowing anything. So each question is sent in four rotations: the correct answer sits on A, then B, then C, then D, with the other three options moved in lockstep. Each rotation is repeated 10 times with a byte-identical request, giving 200 requests and 20 determinism groups.

The result is not a position effect. Across 200 requests the letters were chosen 50/50/50/50 — perfectly flat — and accuracy is 20% at every rotation. The model tracks the content of each option, not its position. It simply prefers the wrong content, and prefers it consistently.

RotationNAccuracyTimes that letter chosen
rotation 050
20%
50
rotation 150
20%
50
rotation 250
20%
50
rotation 350
20%
50
Confidence binNMean confAccuracy %State
0.4–0.5130.4669warn
0.5–0.6390.5521crit
0.6–0.7380.648crit
0.7–1.01100.8318crit

The calibration table is the uncomfortable one: the 110 most confident answers (mean 0.83) are right 18% of the time, while the least confident bin is right 69%. On this quiz confidence points the wrong way — high confidence marks the questions where the plausible-sounding distractor wins.

ONE REQUEST, VERBATIMq4_post_comments.rot0.r00 · 249 MS · 662 TOK IN

The one it gets right. The options are sent verbatim as the four criteria — no hints, no rubric, no reference text. Jev answers B at 0.78. The same dependency, asked as an open five-way classification instead of a multiple choice, is on the Parnas tab.

REQUEST · POST api.typesafe.ai/v1/systemoneAUTHORIZATION: BEARER …
{
  "state": {
    "context": "Consider the following code that you might find in a blog or news website.",
    "code": "class Post {\n  private comments: Comment[];\n  constructor(\n    private id: number,\n    private content: string,\n    private author: string,\n    comments: Comment[]\n  ) {\n    this.comments = [...comments]; // Create a shallow copy of the comments array\n  }\n  getId(): number { return this.id; }\n  getContent(): string { return this.content; }\n  setContent(content: string): void { this.content = content; }\n  getAuthor(): string { return this.author; }\n  getComments(): Comment[] {\n    return [...this.comments]; // Return a shallow copy of the comments array\n  }\n  addComment(c: Comment): void {\n    this.comments.push(c);\n  }\n}",
    "question": "Which of the following proposed changes is most likely to improve the maintainability of this code and the rest of the system? Note that this code may contain other flaws not listed among these choices."
  },
  "model": "jev-latest",
  "questions": {
    "answer": {
      "type": "choice",
      "instructions": "You are taking a software design quiz. `context` introduces the situation, `code` is the snippet under discussion and `question` is what is being asked. Each criterion below is one of the four offered answers, quoted verbatim. Pick the single best answer. Judge the answers on software design merit, not on how they are worded or ordered.",
      "criteria": {
        "A": "The id of a Post should be a hash of its content and author",
        "B": "The Post class should not contain a list of Comments",
        "C": "The comments list should not be copied",
        "D": "The code should not assume that all Posts have authors"
      }
    }
  }
}
RESPONSE · HTTP 200jev-1.13.0
{
  "model": "jev-1.13.0",
  "answers": {
    "answer": {
      "type": "choice",
      "choice": "B",
      "confidence": 0.7,
      "probabilities": {
        "D": 0.14,
        "B": 0.78,
        "A": 0.01,
        "C": 0.07
      }
    }
  },
  "usage": {
    "input_tokens": 662,
    "output_tokens": 45
  }
}
PER QUESTION40 REQUESTS EACH · LETTERS SHOWN ARE THE ORIGINAL QUIZ LETTERS
QuestionPrincipleTruthJevAccuracyp(answer)σConfidenceState
q1_sanitizeimplementation/interface distinctionanswer Bpicked C
0%
0.23±0.030.61wrong
q2_rectanglePlain English Test / Representable-Valid Principleanswer Apicked C
0%
0.08±0.040.88wrong
q3_display_typealgebraic refactoring / a data structure wants outanswer Bpicked C
0%
0.04±0.020.84wrong
q4_post_commentsParnas Subset Criteriaanswer Bpicked B
100%
0.64±0.140.64right
q5_xor_digeststhe design of software is a thing apartanswer Dpicked A
0%
0.06±0.040.61wrong

"Jev" is the option it picked, un-rotated back to the original quiz letter. In every one of the 200 requests, the pick was the same original option — the rotation moved the letter, not the answer.

The four wrong answers have a shape. On q1 it prefers "Option 2 because it's more clear" over the correctness argument; on q2 an IRectangle interface over making illegal states unrepresentable; on q3 named constants over the algebraic refactor; on q5 the reusable-helper answer over "the design of software is a thing apart". Each losing pick is the conventional-wisdom answer — the one that sounds like good practice and that the quiz was built to catch.

q1_sanitize IMPLEMENTATION/INTERFACE DISTINCTION · wrong · 0 of 40

As part of a large application, you are reading an input and need to sanitize it by escaping all single quotes. Which of these two options is better and why? Assume both do the exact same thing.

Option 1: x = sanitize(readInput());
Option 2: x = readInput().replaceAll("'", "\\'");

Multiple benefits of your preferred option may apply. Choose the best answer.

A
Option 1 because it's shorter
0.12
B
Option 1 because option 2 is just plain incorrect, even though it always works
0.23ANSWER
C
Option 2 because it's more clear what it does
0.61JEV
D
Option 2 because it's more efficient, avoiding the function call overhead
0.03

WHY B IS THE ANSWER
The idea of a sanitized string and the idea of a string with escaped quotes are different ideas that happen to share an implementation. Option 2 can only be known to work by relying on a secret of the sanitize module, so it is incorrect as written.

q2_rectangle PLAIN ENGLISH TEST / REPRESENTABLE-VALID PRINCIPLE · wrong · 0 of 40

Consider this Rectangle type as you might find in a GUI framework.

class Rectangle {
  constructor(
    private x: number,
    private y: number,
    private width: number,
    private height: number
  ) {}
  getArea(): number {
    return this.width * this.height;
  }
  getPerimeter(): number {
    return 2 * (this.width + this.height);
  }
}

Which of the following is mostly likely to be a design improvement to this class?

A
The constructor should take two parameters, not four
0.08ANSWER
B
The application should pass around arrays of integers instead of the Rectangle type to avoid the overhead of classes
0.00
C
Rectangle should be made to be an implementation of an IRectangle interface
0.88JEV
D
The class should cache the area and perimeter instead of computing them anew each time
0.04

WHY A IS THE ANSWER
Asked in plain English what data defines a rectangle, people say a corner and a dimension, not four integers. Taking a Point and a Dimension matches that and turns 23 of the 24 argument orderings into type errors.

q3_display_type ALGEBRAIC REFACTORING / A DATA STRUCTURE WANTS OUT · wrong · 0 of 40

Here is a snippet of code you might find in an ecommerce website. Think about what you would do to generally improve this code without changing the behavior.

public setProductDisplayType(displayType: string): void {
    if (displayType === "compact") {
        this.maxDescriptionLength = 140;
        this.numSimilarProductsToShow = 2;
        this.numReviewsToShow = 1;
    } else if (displayType === "detailed") {
        this.maxDescriptionLength = 500;
        this.numSimilarProductsToShow = 5;
        this.numReviewsToShow = 7;
    }
}

Try to think of the best refactoring yourself before looking at the answers. We are deliberately not showing you full detail in the answers below, for fear that showing you the possible solutions would make this question too easy. But if you can think up the best refactoring, then it should be clear which of the answers below is the best match.

A
The best refactored version does the equivalent of looking up a COMPACT_PRODUCT_WIDTH in a hardcoded global hash table
0.12
B
The best refactored version contains some form of the triple (140, 2, 1)
0.04ANSWER
C
The best refactored version replaces the hardcoded strings with constants COMPACT and DETAILED, defined either as String constants or as an enum
0.84JEV
D
The best refactored version uses a switch statement instead of an if-statement
0.00

WHY B IS THE ANSWER
The three numbers travel together, so they are one configuration object. Passing that object is the only answer that truly removes the conditional instead of renaming it.

q4_post_comments PARNAS SUBSET CRITERIA · right 100%

Consider the following code that you might find in a blog or news website.

class Post {
  private comments: Comment[];
  constructor(
    private id: number,
    private content: string,
    private author: string,
    comments: Comment[]
  ) {
    this.comments = [...comments]; // Create a shallow copy of the comments array
  }
  getId(): number { return this.id; }
  getContent(): string { return this.content; }
  setContent(content: string): void { this.content = content; }
  getAuthor(): string { return this.author; }
  getComments(): Comment[] {
    return [...this.comments]; // Return a shallow copy of the comments array
  }
  addComment(c: Comment): void {
    this.comments.push(c);
  }
}

Which of the following proposed changes is most likely to improve the maintainability of this code and the rest of the system? Note that this code may contain other flaws not listed among these choices.

A
The id of a Post should be a hash of its content and author
0.01
B
The Post class should not contain a list of Comments
0.64ANSWER · JEV
C
The comments list should not be copied
0.10
D
The code should not assume that all Posts have authors
0.25

WHY B IS THE ANSWER
A useful subset of the program has posts but no comments, so the Post class must not depend on the Comment class. Copying the list is correct, not a flaw.

q5_xor_digests THE DESIGN OF SOFTWARE IS A THING APART · wrong · 0 of 40

Following is code you might find in a distributed system. A digest in this context is a hash of information relating to the cluster. This function is used to simultaneously compare an unordered set of objects (files, messages, etc) against a checksum. It first combines the digests of each object by taking their XOR, and then compares the result to the target checksum, also using XOR. Note that, while there are many disadvantages to using XOR as a digest, this example is adapted from a real codebase where those were deemed not important.

function compareDigestsToTarget(digests: number[], target: number): boolean {
  let combinedDigest = 0;
  for (const curDigest of digests) {
    combinedDigest = combinedDigest ^ curDigest;
  }
  const difference = combinedDigest ^ target;
  return difference === 0;
}

Choose the best of the following proposed code improvements. There may be other useful improvements not named.

A
The target parameter should be merged into the digests[], so that line 8 can be removed
0.61JEV
B
The code should be optimized to have one less loop iteration, initializing combinedDigest to digest[0]
0.10
C
The two uses of the XOR (^) operator should be extracted into a function
0.23
D
The two uses of the XOR (^) operator should be extracted into two different functions
0.06ANSWER

WHY D IS THE ANSWER
The two XORs have the same implementation but different specifications - combining digests and comparing digests. Identical code with different specs must not be merged.

EVERY RUNCLICK A ROW FOR THE FULL DISTRIBUTION OF THAT RUN
IDQuestionShown answerChosep(answer)p(chosen)Rot
WHAT THIS MEANS FOR THE OTHER THREE TABSREAD THIS BEFORE TRUSTING A BENCHMARK NUMBER

The authored benchmarks name their classes and describe them in the question criteria. "Overloaded means one representation carries two meanings, for example …" is most of the answer. This quiz names nothing: it shows two snippets and four opinions, and the wrong opinions are the ones a competent engineer would defend in review. That gap — 88% and 97% on the authored tests, 20% here — is the honest headline of this whole site.

What survives the gap is narrow and real: given an explicit taxonomy and one snippet, Jev sorts code into that taxonomy quickly, cheaply and repeatably, and it does not cry wolf on clean code. What does not survive is unprompted design judgment. Do not put it in front of a diff and ask "is this good?".

One more caution about this tab specifically: five questions is a tiny sample, and a published quiz may well have appeared in training data — which would make 20% a floor, not a ceiling, and makes the 20% harder to explain away, not easier.