The three tabs to the left are benchmarks written for this model, and it scores 59%, 88% and 98% on them. This tab is somebody else's quiz, written to catch working engineers. Each of the five questions was sent 40 times, with the correct answer rotated through all four letters. 1 of 5 came back right, the same 1 every single time.
No — but the answer never changes. The same request was sent 10 times in a row, 20 times over. In 20 of 20 groups the model picked the same letter in all 10 runs. In 0 of 20 groups the four probabilities came back identical. So the choice is stable and the numbers behind it wobble, by at most 0.034 standard deviation within a group. Ranking never flips; the third decimal is noise.
Where confidence really moves is the letter. On q4_post_comments — the only question it gets right — the probability on the correct answer runs from 0.46 to 0.80 depending on which of A/B/C/D that answer is printed on, a swing of 0.35. The pick is unaffected. Read the confidence as a rank within one request, never as a number you can compare across requests or threshold on.
Practical consequence: repeats buy nothing. Majority vote over 10 runs scores 20%, exactly the same as a single run. That is why every other tab on this site sends each case once.
| Question | p(answer) · rot 0 | rot 1 | rot 2 | rot 3 | Swing | Worst run-to-run σ |
|---|---|---|---|---|---|---|
| q1_sanitize | 0.20B | 0.27C | 0.23D | 0.23A | 0.06 | ±0.032 |
| q2_rectangle | 0.15A | 0.06B | 0.05C | 0.04D | 0.11 | ±0.020 |
| q3_display_type | 0.06B | 0.03C | 0.01D | 0.04A | 0.04 | ±0.014 |
| q4_post_comments | 0.80B | 0.58C | 0.46D | 0.73A | 0.35 | ±0.034 |
| q5_xor_digests | 0.03D | 0.12A | 0.05B | 0.03C | 0.09 | ±0.011 |
Small letter under each number is where the correct answer was printed in that rotation. The swing column is the spread across letters; the last column is the spread across 10 identical requests.
The quiz. Five TypeScript multiple-choice questions from the Mirdin software design self-assessment, transcribed verbatim — context, code, question and all four options. Nothing was reworded. The four options for each question become the four criteria of a single choice question, which is the same primitive the other three tabs use.
The rotation design. A model that likes the letter C scores 25% on a quiz where the answer is always C, without knowing anything. So each question is sent in four rotations: the correct answer sits on A, then B, then C, then D, with the other three options moved in lockstep. Each rotation is repeated 10 times with a byte-identical request, giving 200 requests and 20 determinism groups.
The result is not a position effect. Across 200 requests the letters were chosen 50/50/50/50 — perfectly flat — and accuracy is 20% at every rotation. The model tracks the content of each option, not its position. It simply prefers the wrong content, and prefers it consistently.
| Rotation | N | Accuracy | Times that letter chosen |
|---|---|---|---|
| rotation 0 | 50 | 20% | 50 |
| rotation 1 | 50 | 20% | 50 |
| rotation 2 | 50 | 20% | 50 |
| rotation 3 | 50 | 20% | 50 |
| Confidence bin | N | Mean conf | Accuracy % | State |
|---|---|---|---|---|
| 0.4–0.5 | 13 | 0.46 | 69 | warn |
| 0.5–0.6 | 39 | 0.55 | 21 | crit |
| 0.6–0.7 | 38 | 0.64 | 8 | crit |
| 0.7–1.0 | 110 | 0.83 | 18 | crit |
The calibration table is the uncomfortable one: the 110 most confident answers (mean 0.83) are right 18% of the time, while the least confident bin is right 69%. On this quiz confidence points the wrong way — high confidence marks the questions where the plausible-sounding distractor wins.
The one it gets right. The options are sent verbatim as the four criteria — no hints, no rubric, no reference text. Jev answers B at 0.78. The same dependency, asked as an open five-way classification instead of a multiple choice, is on the Parnas tab.
{
"state": {
"context": "Consider the following code that you might find in a blog or news website.",
"code": "class Post {\n private comments: Comment[];\n constructor(\n private id: number,\n private content: string,\n private author: string,\n comments: Comment[]\n ) {\n this.comments = [...comments]; // Create a shallow copy of the comments array\n }\n getId(): number { return this.id; }\n getContent(): string { return this.content; }\n setContent(content: string): void { this.content = content; }\n getAuthor(): string { return this.author; }\n getComments(): Comment[] {\n return [...this.comments]; // Return a shallow copy of the comments array\n }\n addComment(c: Comment): void {\n this.comments.push(c);\n }\n}",
"question": "Which of the following proposed changes is most likely to improve the maintainability of this code and the rest of the system? Note that this code may contain other flaws not listed among these choices."
},
"model": "jev-latest",
"questions": {
"answer": {
"type": "choice",
"instructions": "You are taking a software design quiz. `context` introduces the situation, `code` is the snippet under discussion and `question` is what is being asked. Each criterion below is one of the four offered answers, quoted verbatim. Pick the single best answer. Judge the answers on software design merit, not on how they are worded or ordered.",
"criteria": {
"A": "The id of a Post should be a hash of its content and author",
"B": "The Post class should not contain a list of Comments",
"C": "The comments list should not be copied",
"D": "The code should not assume that all Posts have authors"
}
}
}
}{
"model": "jev-1.13.0",
"answers": {
"answer": {
"type": "choice",
"choice": "B",
"confidence": 0.7,
"probabilities": {
"D": 0.14,
"B": 0.78,
"A": 0.01,
"C": 0.07
}
}
},
"usage": {
"input_tokens": 662,
"output_tokens": 45
}
}| Question | Principle | Truth | Jev | Accuracy | p(answer) | σ | Confidence | State |
|---|---|---|---|---|---|---|---|---|
| q1_sanitize | implementation/interface distinction | answer B | picked C | 0% | 0.23 | ±0.03 | 0.61 | wrong |
| q2_rectangle | Plain English Test / Representable-Valid Principle | answer A | picked C | 0% | 0.08 | ±0.04 | 0.88 | wrong |
| q3_display_type | algebraic refactoring / a data structure wants out | answer B | picked C | 0% | 0.04 | ±0.02 | 0.84 | wrong |
| q4_post_comments | Parnas Subset Criteria | answer B | picked B | 100% | 0.64 | ±0.14 | 0.64 | right |
| q5_xor_digests | the design of software is a thing apart | answer D | picked A | 0% | 0.06 | ±0.04 | 0.61 | wrong |
"Jev" is the option it picked, un-rotated back to the original quiz letter. In every one of the 200 requests, the pick was the same original option — the rotation moved the letter, not the answer.
The four wrong answers have a shape. On q1 it prefers "Option 2 because it's more clear" over the correctness argument; on q2 an IRectangle interface over making illegal states unrepresentable; on q3 named constants over the algebraic refactor; on q5 the reusable-helper answer over "the design of software is a thing apart". Each losing pick is the conventional-wisdom answer — the one that sounds like good practice and that the quiz was built to catch.
As part of a large application, you are reading an input and need to sanitize it by escaping all single quotes. Which of these two options is better and why? Assume both do the exact same thing.
Option 1: x = sanitize(readInput());
Option 2: x = readInput().replaceAll("'", "\\'");
Multiple benefits of your preferred option may apply. Choose the best answer.
WHY B IS THE ANSWER
The idea of a sanitized string and the idea of a string with escaped quotes are different ideas that happen to share an implementation. Option 2 can only be known to work by relying on a secret of the sanitize module, so it is incorrect as written.
Consider this Rectangle type as you might find in a GUI framework.
class Rectangle {
constructor(
private x: number,
private y: number,
private width: number,
private height: number
) {}
getArea(): number {
return this.width * this.height;
}
getPerimeter(): number {
return 2 * (this.width + this.height);
}
}
Which of the following is mostly likely to be a design improvement to this class?
WHY A IS THE ANSWER
Asked in plain English what data defines a rectangle, people say a corner and a dimension, not four integers. Taking a Point and a Dimension matches that and turns 23 of the 24 argument orderings into type errors.
Here is a snippet of code you might find in an ecommerce website. Think about what you would do to generally improve this code without changing the behavior.
public setProductDisplayType(displayType: string): void {
if (displayType === "compact") {
this.maxDescriptionLength = 140;
this.numSimilarProductsToShow = 2;
this.numReviewsToShow = 1;
} else if (displayType === "detailed") {
this.maxDescriptionLength = 500;
this.numSimilarProductsToShow = 5;
this.numReviewsToShow = 7;
}
}
Try to think of the best refactoring yourself before looking at the answers. We are deliberately not showing you full detail in the answers below, for fear that showing you the possible solutions would make this question too easy. But if you can think up the best refactoring, then it should be clear which of the answers below is the best match.
WHY B IS THE ANSWER
The three numbers travel together, so they are one configuration object. Passing that object is the only answer that truly removes the conditional instead of renaming it.
Consider the following code that you might find in a blog or news website.
class Post {
private comments: Comment[];
constructor(
private id: number,
private content: string,
private author: string,
comments: Comment[]
) {
this.comments = [...comments]; // Create a shallow copy of the comments array
}
getId(): number { return this.id; }
getContent(): string { return this.content; }
setContent(content: string): void { this.content = content; }
getAuthor(): string { return this.author; }
getComments(): Comment[] {
return [...this.comments]; // Return a shallow copy of the comments array
}
addComment(c: Comment): void {
this.comments.push(c);
}
}
Which of the following proposed changes is most likely to improve the maintainability of this code and the rest of the system? Note that this code may contain other flaws not listed among these choices.
WHY B IS THE ANSWER
A useful subset of the program has posts but no comments, so the Post class must not depend on the Comment class. Copying the list is correct, not a flaw.
Following is code you might find in a distributed system. A digest in this context is a hash of information relating to the cluster. This function is used to simultaneously compare an unordered set of objects (files, messages, etc) against a checksum. It first combines the digests of each object by taking their XOR, and then compares the result to the target checksum, also using XOR. Note that, while there are many disadvantages to using XOR as a digest, this example is adapted from a real codebase where those were deemed not important.
function compareDigestsToTarget(digests: number[], target: number): boolean {
let combinedDigest = 0;
for (const curDigest of digests) {
combinedDigest = combinedDigest ^ curDigest;
}
const difference = combinedDigest ^ target;
return difference === 0;
}
Choose the best of the following proposed code improvements. There may be other useful improvements not named.
WHY D IS THE ANSWER
The two XORs have the same implementation but different specifications - combining digests and comparing digests. Identical code with different specs must not be merged.
| ID | Question | Shown answer | Chose | p(answer) | p(chosen) | Rot |
|---|
The authored benchmarks name their classes and describe them in the question criteria. "Overloaded means one representation carries two meanings, for example …" is most of the answer. This quiz names nothing: it shows two snippets and four opinions, and the wrong opinions are the ones a competent engineer would defend in review. That gap — 88% and 97% on the authored tests, 20% here — is the honest headline of this whole site.
What survives the gap is narrow and real: given an explicit taxonomy and one snippet, Jev sorts code into that taxonomy quickly, cheaply and repeatably, and it does not cry wolf on clean code. What does not survive is unprompted design judgment. Do not put it in front of a diff and ask "is this good?".
One more caution about this tab specifically: five questions is a tiny sample, and a published quiz may well have appeared in training data — which would make 20% a floor, not a ceiling, and makes the 20% harder to explain away, not easier.