Benchmarking LLMs for GenUI: Speed, Schema Conformance, and Human Taste

Generative UI is transforming how Flutter developers build interfaces, but choosing the right large language model for the job is far from straightforward. Very Good Ventures engineer Brian Egan recently published a detailed benchmark of multiple LLMs for GenUI workloads, measuring not just raw speed but also schema conformance, error rates, and the subjective qualities that no automated test can capture. The results reveal a landscape where the fastest model is not always the best choice.

Egan built the benchmark while developing VGV's GenUI application for Google Cloud Next. Instead of testing through the full app stack including widget trees and animations, he drove the conversation layer directly with the same prompts and parsers the app uses. This headless approach meant the benchmark could run anywhere, including a plain Linux CI runner, without browser or platform setup overhead.

What the Benchmark Measures

For every model tested, the harness tracks four primary metrics:

• Total round-trip time — the wall-clock time from request submission to full response receipt
• Time to first chunk — how quickly the model begins streaming its response, which heavily influences perceived responsiveness in GenUI applications
• Output token count — the verbosity of each model, which directly affects both latency and cost per generation
• Error rate — the percentage of responses that fail schema validation, request errors, hallucinated components, or malformed properties

Each model's output is validated against the same JSON schemas that define VGV's design system catalog. If a component's properties do not match the catalog's schemas, if the model reaches for a component that does not exist, or if the request errors outright, it counts as a failure. This strict approach prevents broken screens from quietly looking like successes, as can happen when the GenUI framework logs the problem and renders anyway.

Surprising Findings in the Data

One of the benchmark's most valuable findings was methodological: running models one after another in a batch introduces significant bias. When Egan hit one provider with a burst of requests, rate limiting kicked in, and whichever model ran later in the batch appeared artificially slow. The fix was a round-robin schedule with one request per model per cycle, reshuffling the order each round so that provider load spreads evenly.

Another finding concerns the thinking mode that many modern LLMs support. Disabling thinking often makes responses significantly faster, but the mechanism varies by provider. Gemini allows dialing the level down, OpenAI's GPT-5 models only go as low as "low," DeepSeek needs a non-standard request field, and Kimi's k2.7-code will not let you turn it off at all. Egan benchmarked each model in both thinking and no-thinking variants where possible, giving developers concrete data on the speed trade-off.

Why Speed Is Not the Whole Story

Despite the rigorous measurement framework, Egan emphasizes that speed and validity alone do not determine the best model for GenUI. Whether a generated interface is clear, well-organized, and presents the right information in the right format ultimately comes down to human taste. The benchmark narrows the field to models that are fast and produce valid UI, but the final decision requires a person to judge whether the experience meets the bar.

Models tested include various configurations of Claude, Gemini, OpenAI's GPT-5 family, DeepSeek, and Kimi. The benchmark code itself was built with Claude Code, which Egan credits for chasing down surprises like the rate-limiting issue that the round-robin schedule solved. However, he notes that a person still owned every decision that mattered — what to measure, what counts as a real failure, and whether the output is actually any good.

For Flutter developers building GenUI applications, this benchmark provides a practical starting point for model selection. The key takeaway is that benchmarking methodology matters as much as the results themselves, and that automated testing can only take you part of the way toward a great generative user experience.