What's in the box?!

Welcome to Product Cocktail, where the takes are as polarizing as a shot of Fernet—but the insights come together like a perfectly crafted daiquiri.

Tuesday, August 4th. 11:07 AM

I type "run smoke test" into the chat bar and hit enter, sitting patiently as Claude Code goes off to do my bidding.

After a few minutes:

The first AI model test for my new app: mission failed, successfully.

Remember that "Cocktail Pokédex" idea from my AI Design Tool Series at the end of June? I'm building it. In public. Follow along here.

I had spent the last day and a half finding 15 cocktail / menu photo pairs, outlining a schema for what data would appear in the app, manually building an answer key for what the app should understand about the cocktails, and building a script to test whether the core AI feature will work: cataloguing cocktail information from photos.

This failure wasn't random.

My original plan — courtesy of Gemini's insights on vibe coding best practices from the top AI-pilled product luminaries — was to do a week of spec, a week of UI refinement, then build the AI pipeline in week three.

Fable had different ideas. I used some of my precious Claude credits to run this plan by the proverbial Father, Son, and Holy Spirit of AI, and the first thing that jumped out: I was missing a huge de-risking opportunity with this order of operations.

Spending three weeks on commodity plumbing (frontend: Next.js / database: Supabase / payments: Stripe) only to potentially find out that existing computer vision AI might fail at recognizing cocktail metadata is a huge waste of time.

Instead, I spent two days on a technical spike to answer a question: does vision extraction work well enough to justify building my cocktail pokédex?

This isn't productivity theater either, this is a real speed-to-market win like I discussed back in May.

Two years ago, at HBO Max, that's roughly the amount of time it would have taken to get the engineering handover scheduled, before any work started. And that's saying nothing about the jockeying with fellow PMs to win a slot in the Growth Product team's prioritization queue. (To be clear, this is partly an indictment of engineering:product organizational structure, not purely an AI cheerleading story.)

Interestingly, most of my time in those two days was spent sourcing test data and drafting the rubric. The compute wasn't the bottleneck, judgment was — although "accept edits" mode on Claude Code might actually make me reach for the bottle.

In this particular AI development scenario, there is a right answer. I'm searching for a deterministic result from a non-deterministic technology.

In order to effectively test the model, I needed legitimate test data (cocktail and menu photo pairs) and an answer key to grade against.

Another recommendation from Fable: if you score the outputs after seeing them, you'll produce a grade inflated "eh, close enough" result. In other words, run the test at the School of Hard Knocks, not Harvard.

I hard coded this into the script to avoid accidentally burning my test data. It refused to run until the source of truth files were marked as completed.

This produced real product insight: I ended up merging several vocabulary options in the schema. Shot and shooter. Crushed and pebble ice. Different, but functionally hard to tell apart in a grainy photo. Keeping them separate would have manufactured errors that taught me nothing.

Sometimes "I don't know" is the right answer.

Look again at that Mistah Shōjō photo above.

Quick, what's the ice type? Whiskey cube? Standard? Pebble? Spear? None?

You wouldn't know from that picture. The foam and the opacity of the cocktail are hiding it.

A model that says unknown is behaving better than one that randomly guesses something.

Even if the answer is binary (e.g. is this a drink the bar invented, or a canonical recipe like a Mai Tai?), the evidence isn't. Nothing in a photograph of a drink says whether that bar invented it.

I ran every drink three ways: menu photo and drink photo, menu only, drink only. Parentheses are how often the model abstained.

Wrong and abstained earn the same grade when scoring, but one codifies incorrect metadata, the other asks the user to fill in the blank.

Taking away the opt-out doesn't make the model more honest, it makes the scoreboard less honest.

The schema didn't give the model a way to say "I don't know."

The model kept trying to report that a field was undeterminable and the schema kept rejecting the record for saying so.

The backend field that logged the proof of each data point (evidence) only allowed menu | photo | derived | recalled as responses, presuming that the field has a source. Without the menu, price has none. The entire call failed, including the fields it did read correctly.

The crashes weren't the problem. The most costly defect didn't crash at all.

The "house original?" field was a boolean, with no abstention path, so it never abstained, it guessed. It was right about half the time, and it decides whether a drink can carry a canonical recipe at all — get it wrong and you attach a stranger's Mai Tai spec to a drink the bar invented.

The ones that crashed cost me nothing but time. The dangerous one confidently returned a plausible number.

After cleaning up the schema bugs, I realized that 23% of the menu-included runs transcribed a completely different drink.

When I split the test results by whether the right entry was found, it had an extraordinary impact on the overall correctness. (No shit.)

One wrong pick corrupts seven fields at once: name, ingredients, price, section, spirit base, family, and style.

Two pairs were misidentified on all four attempts: both models (Haiku 4.5 and Opus 5), both conditions (menu only or menu + cocktail). Turns out, even super intelligence can't read minds.

This wasn't a model problem, this was a product problem.

A cocktail menu entry.

There's an entire subreddit dedicated to shaming "useless red circles" on screenshots. This is the opposite of that.

After seeing the mai tai issue, I hand-marked all 15 menus with a red box around the target entry.

This one step entirely killed the wrong drink selection failure mode. Menu only: 28% → 0%. Menu plus cocktail photo: 19% → 0%.

Splitting that by model is the part that stopped me. Haiku picked the wrong entry on 21% of unmarked menus. Opus — nine times the price per call — picked wrong on 23%. On menu-only captures it was 20% against 33%, with the expensive model doing worse. Paying more did not buy a better read of the menu. A red box drawn by hand took both to zero.

This also drastically improved performance across key fields:

Turns out, attention (to detail) is all you need.

The natural question after solving menu selection: does the model still need the drink photo?

The photo itself was never in question — the Dex is a visual product, and nobody wants to scroll a collection of dim menu crops. That said, I don't have to send an image to the API to show it in the app. On Haiku, the drink photo runs about a sixth of a cent per capture: 16% of the call, on every capture, forever.

Claude Code surfaced the idea while interpreting the results — flagged as the "open thread most likely to change the ending" of this newsletter, on the strength of menu_only_marked 83% vs both_marked 81% for Opus.

The number was real. The comparison wasn't — and the part that's on me isn't the table, it's that I believed it. Claude wrote the grading script and generated the report; I read the number and wrote it into my outline for this issue as the thing most likely to change the ending. I never asked what was in each denominator.

That 83/81 came out of an "answerable cells only" table, which answers a different question: how good is the model, with the structurally impossible stuff filtered out. It drops unanswerable cells per condition — menu-only averaged over 10 fields, both over 14. The four extra (glassware, ice, color, garnish) are the hardest in the set. Menu-only didn't win. It got asked easier questions.

Count every field, including the ones a menu can't answer, and it isn't close:

The cocktail photo even improves fields it can't literally see: method 42→64% and family 60→73%. A photo reliably tells you whether a drink was shaken, and the glass narrows the family.

Normalizing is for measuring the model. Deciding what to build means counting everything the input can't answer, because that's what matters to the user.

Nobody would have reviewed it. This isn't a feature cut — no spec change, no design review, no ticket. It's one person deleting one argument from one API call on a Tuesday afternoon to save a dollar per six hundred drinks. The photo still shows up in the Dex. The app still looks right. Four fields just silently get worse. A regression with no symptom is worse than a broken feature — a broken feature at least tells you it's broken.

Even casual readers know that I'm no fan of yeeting money into the abyss for the sake of AI. Beyond answering "will this work?" I wanted to understand what I was getting for my money.

I looked at the accuracy of non-transcribed fields, because those are the ones the user can't directly check. A wrong drink name is obvious, a botched spec is not.

Note: Sonnet only ran the marked conditions.

Unfortunately for my inference costs, the performance is linear. Each additional point of accuracy costs about half a cent per API call.

Furthermore, Haiku's weakness is specific: the drink photo processing is much poorer. Across all 15 drinks, Haiku correctly identified 45% of glassware/ice/color/garnish attributes, with Sonnet and Opus at 72%.

The failure is concentrated, however. Haiku and Opus perform equally poorly on color, but there's a real gap on ice, driving most of the -15pp accuracy gap there alone.

If I ship Haiku, my design note (to myself) will be: put a hard review gate on ice_type and a soft review on garnish. Color requires revisiting the schema to understand if the vocabulary is too fine or grading is too strict.

The spike was supposed to answer "does the AI work?" It also rewrote the UI backlog and data model, mostly from looking at the answer key.

None of these insights came from PRD iteration with stakeholders. They came from $9.55 and 180 API calls — about half the cost of Mistah Shōjō after tax and tip — and a hand-written answer key.

Four things went wrong in this spike. The model picked the wrong drink off the menu. The schema wouldn't let it say "I don't know." A summary table told me to stop sending the cocktail photo. Haiku turned out to be the weakest on the fields that the photo exists for.

Three of those things had nothing to do with the model. They were fixed by a rectangle, a third enum value, and a denominator — and on the first one, the expensive model was actively worse.

The fourth is different. That's a real capability gap with a price tag on it. Sonnet and Opus read a glass and its contents better than Haiku, and they'll keep doing that on every call. I can buy the fix or I can build a review gate around it. That's a purchasing decision, not a design one.

It's a less tidy conclusion than "the model doesn't matter" and closer to the job. Some of your problems are worth paying to solve. Most of mine weren't. I went in asking whether the AI was good enough, and three times out of four, the answer came back as a product question.

In the coming weeks I'll cover the other product insights that came out of this spike: measurement and product vision. Beyond that, I'll keep iterating and check back in when there's something worth showing. If something pressing comes up, I'll pause the build-in-public series to cover it (surveillance pricing, anyone?).

One thing I deliberately didn't test: model choice beyond Anthropic. I built the harness against the Claude API because that's what Claude Code defaulted to and what I know, having built my agent on Sonnet 4.6. Whether GPT-5.6 or Gemini is a better bang for the buck is an open question, and a future issue.

Send me questions, feedback, and cocktail recipes:
[email protected].

_{Icons made by Icongeek26 from www.flaticon.com.}