The Center for Models Who Can't Read Good

Welcome to Product Cocktail, where the takes are as polarizing as a shot of Fernet—but the insights come together like a perfectly crafted daiquiri.

I don't subscribe to the tokenmaxxing AI development theater philosophy where you throw the most expensive AI model at a problem and play chicken with your credit card company. Real ones are familiar with my stance here.

That's why my going in design standard for Proof—the AI cocktail logging app I'm building—was to ship with the cheapest model that could do the job.

In the first issue of the Proof (fka Cocktail Pokedex) build log, I hinted that Haiku would involve a trade-off.

That issue asked whether the model was good enough. I ran another test to answer something the first spike couldn't: whether the review gate catches the model when it isn't. The gate only emerged after I did the initial spike and wrote that newsletter.

Putting the updated prompt against the fifteen photo pairs asks: how many of the nine fields the review screen puts in front of the drinker need to be touched before the specimen can be filed?

The result: both frontier models (Sonnet 5 / Opus 5) landed on a median of one field to correct, with 3 of the captures needing no update. Haiku is significantly worse, with a median of 2 fields and only 1 clean capture. The aggregate view confirmed my original read that Haiku produces a worse result but is fixable with a review gate.

On Sonnet and Opus, the three most important fields: name, price, and ingredients are 14/15 or better with zero silent errors, meaning an incorrect field coded as "high confidence." The real seam between the frontier models appears in two places.

The first is visual attribute recognition—ice_type or garnish "observed" from the cocktail photo. This is the clearest demonstration of the image processing quality gap between the models.

Within that tier it's lopsided. Glassware is close to solved on all three — a coupe looks like a coupe. It's garnish where everyone loses, and ice_type where only Haiku does.

The second is a different problem: method scores 10/15 on all three models. That looks like a field nobody can do, until you look at what the misses actually are.

Opus adds methods beyond what my key listed—five times—and is never actually wrong. Some are debatable (plenty of bartenders build an Old Fashioned in the glass rather than stirring it in a mixing glass) and some are more correct than my key (an egg white drink getting a dry shake when I'd only written shaken.) Sonnet adds three and gets two genuinely wrong. Haiku adds one and gets four wrong, including dropping rinse from a rinsed-and-stirred drink at high confidence.

Same number, three different behaviors, ordered exactly by capability. The aggregate said all three were equally bad at method, and not one of them were "bad" in the same way.

Haiku misses on both of these, plus the transcribed tier that feeds the derived fields—two misses each on name and price. Haiku's image-to-text transcription (transcribed) is about as good as Opus' identification of attributes from a photo. The frontier models fail in mostly one place; the cheap one fails everywhere.

The transcribed and observed tiers both arrive as photographs—the model is sent a cropped menu entry and the cocktail photo as images. The difference is that on the former, the answer to the test is right there, the model just needs to read it.

The derived tier applies a rule to those same words. spirit_base scans the ingredient list in order, then family and method follow from what the drink is. All six of these fields can theoretically be answered by the menu photo alone.

The observed tier looks at a cocktail photo and tries to guess what glassware, ice_type, and garnish are based on what it looks like. The frontier models have solved the first two tiers and are still losing a third of the last one.

If the ingredient list comes back slightly incorrect, it's a minor correction on the review page. If it's significantly wrong, that's probably a wrong entry or an unreadable shot, and that's where a retake comes in handy.

That sets the bar for capture rather than for the review screen. The transcription must be overwhelmingly correct most of the time because the consequence is someone standing at a bar buried in their phone while their drink gets warm. (This is why I stopped using Untappd.)

When I originally built the confidence scoring, the review gate triggered on fields that sit below a high so it can only catch errors the model is unsure about.

The two fields I originally considered review-gating out of the first spike were ice_type and garnish. On Haiku, those fields are incorrect and scored high confidence on ice over half the time and on garnish two thirds of the time.

Across the nine fields, Haiku files 27 silent errors, 20% of the fields across the test data.
The review gate is a deadbolt on a door that's already open. It's not stopping anything from getting inside.

I said this was a purchasing decision, not a design one. It was neither. You can't out-design Haiku's model limitations with a review gate. A gate only catches what the model is unsure about and on the two fields I wanted to gate, Haiku isn't unsure. It's confidently wrong.

I built Proof's schema to identify cocktail "species" on venue + name. The name is editable at capture and never afterwards. A confidently wrong name that the drinker doesn't catch isn't annoying data cleanup, it's a permanent data quality issue that's flipping you off every time you scroll past it in the catalogue.

Haiku botched the names on two of the fifteen cocktails. Not ideal.

Currently, Haiku is $0.95 per 100 captures against Sonnet's $5.51. The $4.56 saved is paid by over 10% of species names being read incorrectly. Who in their right mind would ship a product like that? (Google, apparently, with AI Overviews.)

Incorrect transcription one out of every ten times is not a compelling AI product, it's a cocktail-themed proofreading quiz.

The split between Haiku and frontier is "can't read good" and my Spidey Senses tell me that would be an experience-breaking UX for an app that demands precision and speed.

Sonnet and Opus go drink-for-drink on the headline workload. They diverge where the errors land: Sonnet clocks 2% of fields confidently wrong, with Opus at 5%, compared to Haiku's 20%. All three of Sonnet's silent errors were garnish, with none on the other fields.

The apparent price is Sonnet's false positive rate: 31% vs. Opus' 24%. Flagging 31% of correct fields as potentially wrong reads like a tax on the user, but this is driven by the model's tendency to hesitate. That also means it files almost nothing wrong without asking. This is the correct calibration.

Whether the hesitation stops the drinker or merely flags the field is a separate question, and it's a UX consideration. A flagged value that doesn't block is still labeled, carrying the model confidence and whether the drinker changed it—still valid signal for evals.

The product decision is a trade-off between interrupting the user (blocking the filing until fields are confirmed) or risking incorrect medium confidence fields pass through into the user's specimen entry.

Sonnet ships provisionally. The split between frontier models sits on the least consequential fields on the screen, which is an argument for seeking real user feedback.

The test data and rubric were built by one person: me. I knew what fields I was extracting in advance, and then I tuned the prompt against these images. The remaining question to answer is not which model is better on this narrow set, it's which is better on captures taken by someone who has never looked at the schema.

That's why I built a backend before I did any more model testing. Live extraction needs an API key and I can't hard code that into an app I'm going to release into the hands of real users. Any shortcut I took to address that without actually building the auth/database would just have been throwaway work.

What I built this week: magic link sign in (no passwords!). Your specimens stop living on your phone and start living in your account, photographs included. The database itself enforces who can read which records, so your low key bender on Thursday remains a secret.

For the nerds, the stack expanded significantly in the past few days:

Next, I’ll be wiring in extraction with a real model, then I’ll be ready to launch a beta. That's where YOU come in, fellow product and cocktail enthusiasts... (Read the Garnish 😉).

Sign up for the Proof beta!
[email protected].

_{Icons made by Icongeek26 from www.flaticon.com.}