Published November 20, 2026

Free Graders: Why a Passing AI Skill Eval Proves Nothing About Your Flutter Plugin

Very Good Ventures ships fifteen skills with its open source AI Flutter plugin, and twelve more with Wingspan. Each skill is a markdown file of opinions written down for a model to read: use Equatable, name the loading state LoginInProgress, keep event names in the past tense. The team had evidence that the skills worked, and the evidence was that the output had looked fine when a person tried it by hand.

That is not measurement. In a post published on November 16, Dominik Simonik walks through what happened when the team built a 102-case eval suite to replace the guesswork, and why the first thing the suite taught them is that most of their passing checks were proving nothing about the plugin at all.

Routing first, content second

The first question in an eval is whether the model read the skill. Claude Code does not load every skill in a plugin. It reads the short description at the top of each skill file, decides whether to load one, and starts writing. That decision is called routing, and the output looks identical whether it happened or not. Ask for a login bloc with no plugin installed and a good model still returns sealed classes, pattern matching, and sensible names.

So the suite measures instead of assuming. Each check is called a grader and returns pass or fail. The routing grader reads the run's trace, the log of every tool the model called, and looks for a call to the Skill tool naming bloc. Across the three runs behind the post, the skill loaded every time. If that rate drops, the fix is the description, because the description is the only thing the model reads before deciding.

The suite measures with all fifteen skill descriptions loaded, not just the one under test. Fifteen descriptions compete for the same routing decision, and that competition is what a real session presents.

Every case runs twice

The login bloc prompt reads like a ticket. It asks for a LoginBloc handling email and password authentication, a submit event and a logout event, a spinner while the request is in flight, a User object on success, and an error message on failure. It never names the skill, never says Equatable, never mentions sealed classes.

Eight graders score that reply, each with a weight, adding to 10. The case score is the weight that passed divided by 10, and the case passes when it clears a threshold. The runner defaults to 1.0. The team set theirs to 0.8, because two of the graders use a second model as a judge and a judge can mark a correct answer wrong.

Then comes the part that changes how you read every eval result. Each case runs twice: once with the plugin loaded and once with no plugin at all, in a throwaway home directory. The gap between the two scores is reported as delta, and it is the only part of the number the skill can claim. A grader that passes whether or not the plugin is loaded is measuring the model. From inside a passing run, it looks exactly like a grader measuring the skill.

The three grader types each carry a blind spot worth knowing:

  • Regex graders read the reply text. They cannot tell a class name from the same name inside a comment.
  • Tool graders read the trace and count calls. They see what the model did, not what it said it did, which is why routing uses one.
  • Rubric graders hand the reply and a written criterion to a second model, which votes three times so a split decision stays visible instead of disappearing inside a pass.

Four of eight graders were free

In the login case, measured in October 2026 over three runs per arm, the arm with the plugin scored 1.00 every time. The bare model scored 0.50, 0.70, and 0.40, for a mean delta of 0.47.

The headline number is the least interesting part. Four of the eight graders passed in both arms on every run: the sealed event hierarchy, the sealed state hierarchy, past-tense event names, and the LoginInProgress naming convention the skill pins. The bare model produced all four unprompted. The team calls those free graders, and no amount of green in a report makes them evidence.

What the plugin actually added was Equatable. The state class came back extending Equatable with a props getter in the plugin arm, and without it in the bare arm. That single convention is where the measurable contribution of the skill lived in that case.

Reading the per-grader rows rather than the summary is what sent the team back to the suite. One pull request touched graders in seven of the fifteen skills. In four of its cases, graders were passing without the plugin, and reworking them roughly doubled those cases' delta.

There is a subtler accounting trap too. The runner drops a routing grader from both arms by default, because a Skill call can never happen without the plugin and counting it would inflate the gap. The team forces it back in, because the routing weight of 3 exists so a missed skill drags the case below the 0.8 threshold, and a grader the runner ignores cannot drag anything. Leave that line out and the case reads 0.24 instead of 0.47.

The lesson transfers to any Flutter team writing agent instructions, whether the target is Claude Code, Antigravity, or a homegrown pipeline. Three habits from unit testing break here: same input does not produce the same output, a passing test is not necessarily a good test, and red does not always mean a bug in your change. The second one is the expensive one. A suite that only runs with the plugin loaded cannot tell you which half of the score came from you.