Shaken, Not Measured

Welcome to Product Cocktail, where the takes are as polarizing as a shot of Fernet—but the insights come together like a perfectly crafted daiquiri.

It was my third day building Proof. I had just reframed the app from a completionism-oriented checklist to an open world exploration of novel cocktails.

Claude Code confidently told me that the entire point of my app — to remake cocktails at home — was dead on arrival.

I too was confident — that Claude Code had no idea what it was talking about.

Instinctively, I knew that this assertion was wrong. I've used AI models to intuit, tweak, and ideate cocktail specs in the past. There are like 7 - 12 base cocktails, depending on who you ask, and they're literally formulas.

I ran a small test: ten drinks, three models, $0.23. I asked if each of the three models could reconstruct a home spec with only the ingredients, method, and glass, no amounts.

Most of these were my own, unpublished specs: Rolling with Krampus, The Griswold, "Keep the Change, ya Filthy Animal". Riffs to be sure, but the name-to-spec mapping is unpublished, making this a test in deriving amounts, not recalling them from memory.

Interestingly, given an open invitation to embellish the recipe lists ("only if the drink genuinely will not work without something that was not listed"), the models only took it three times — where an ingredient list was actually incomplete. Egg white or an Angostura bitters garnish added to a whiskey sour riff, expressed orange peel to an Old Fashioned.

The initial results came back: 50% - 67% within tolerance. Not great, but potentially a shippable MVP.

One line in the summary caught my attention:

If you've ever looked at a jigger, you can probably predict my response. I literally used the word "insane" when I asked the agent about it.

Assuming this was an ounce measure, I corrected the model, explicitly stating that everything should be returned in jigger-steps (0.25 oz, 0.5 oz, 0.75 oz, etc.).

This wasn't ounces, it was a ratio. The grader divides every amount by the base spirit, creating a normalized ratio for each component (i.e. 0.75 oz / 2.0 oz base = 0.38). The reasoning: the right ratio of ingredients preserves the spirit of the cocktail and should count as a correct answer. Logging a double and then grading 1x ingredient amounts as wrong is nonsensical.

The ratio scoring worked to solve a real problem, but chasing the units found the metric underneath, which was distorting everything built on top of it.

The grader scored ingredient amount accuracy with relative error. In other words, how far off is the amount as a percentage. Data scientists are nodding. Industry vets are confused.

The choice was made when Claude Code built the evaluation harness, automatically.

Relative error scales with pour size. Sweeteners are typically the smallest ingredients in a cocktail (in my data: 0.45 oz mean, compared to 0.68 oz for modifiers like citrus or amaro). In other words, a quarter ounce miss reads ~60% error on a sweetener and ~29% on a modifier. The same error, priced differently because the pour is smaller.

If you score the sweeteners the way a bartender would, in quarter ounces, that 60% error becomes zero. Not one of them was off by more than a quarter ounce.

Based on these results, Claude wrote into the decision log "sweetness is where reconstruction fails, at roughly twice the rate" with a product design edict: make the sweetener an adjustable dial in the reconstruction UI, claiming that "the measurement hands you the design."

The measurement hands you the design, indeed. Except when it's a misread of the data, divorced from reality. This design choice was argued and codified into the project plan without approval. It was withdrawn immediately.

A normalized metric isn't neutral. It encodes a claim about what counts as a big error. This one claimed that 0.25 oz matters more in a small pour than a large one.

Two issues ago, I mentioned that a denominator nearly talked me out of sending the drink photo. That was about what I compared, this is about the ruler. A wrong denominator is an evaluation bug. A wrong metric is a judgement you didn't know you were making.

Across the 36 ingredient predictions, only one missed by more than a quarter ounce. Nineteen were exact matches.

To be clear about what this is: twelve scored amounts per model, one run each. That's enough to tell me a feature isn't dead on arrival. It is not a benchmark.

Every one of these specs is also mine. This measures reconstruction against exactly one bartender's palate. A miss against my spec isn't necessarily a bad drink — it's a drink I wouldn't have poured.

Haiku and Sonnet both went twelve for twelve. Opus missed one. At this sample size that's a coin flip rather than a ranking — the part that matters is that the cheapest model is in the group that missed nothing, at $0.0018 a call.

On a 0.45 oz sweetener, ±25% is about a tenth of an ounce; the step metric allows a quarter ounce — more than twice as much. On the bigger pours the gap narrows: ±25% of 0.68 oz is about 0.17 oz, against the same quarter ounce. It's a looser ruler exactly where it matters, on the smaller (sweet) ingredient amounts.

This isn't a lenient tolerance, it's a practical reality of the tools available. Bartenders aren't using graduated cylinders, they're using quarter-step jiggers or pour counts.

And to be clear, a quarter ounce matters — but it's within a fixable tolerance, not something you dump in the sink and start over.

Claude asserted that the pipeline couldn't deliver on "make at home" for 80% of captures. This test proved it wrong.

A normalized metric encodes a judgment about what counts as a big error. Say that choice out loud before you get your Data Science team involved.

Relative error isn't a mistake. It's the choice of a competent practitioner at the helm of a test result dashboard. If someone handed me these results in a doc, I probably would have signed off on them. It was incorrect because it didn't clock that sweeteners are the smallest pour in a cocktail.

This wasn't an engineering failure, it was a reasonable assumption made without domain context. I'm not sure a PRD closes that gap either, because it isn't obvious until you're looking at the results.

The gap wasn't the metric, it's that a measurement became a written decision without an approval step in between. Having been burned by this, I changed the rules — no more decisions logged without explicit approval.

Every earlier bug in this project was found by staring at the numbers. This one was found by knowing what a jigger looks like. The expertise that catches a measurement error is often not measurement expertise.

I've been porting the design from v0 into usable code for App Store / Play Store builds and making some key UI changes.

I'll share more details soon, particularly the testing that showed me the unfortunate consequence of a five pixel difference in tap location on device.

Send me questions, feedback, and cocktail recipes:
[email protected].

_{Icons made by Icongeek26 from www.flaticon.com.}