How to prove one-tap actually worked
Scans are not conversions, and a lift is not a cause. This is the measurement design to agree before anything gets printed — plus what we won't claim, and why.
What this article is, and isn't
This is a method, not a result. Forage has no published customer case study yet, and we'd rather say so than dress a model up as one. When we have a customer whose numbers came from their own analytics, who signed off the text, and whose measurement design we can show you, we'll publish it and it will look like the standard described below. Until then, the useful thing we can give you is the design — which is worth more anyway, because you'll need it whether or not you work with us.
Nearly every scan-to-buy programme is measured the same way: count the scans, count the orders that followed, put both on a slide. That number is real, and it answers almost nothing worth knowing. It cannot tell you whether the orders were incremental, whether they cost you retail margin, or whether the same revenue would have arrived anyway.
Scans are not conversions
Start by being precise about what each metric in the chain can support.
| Metric | What it tells you | What it can't tell you |
|---|---|---|
| Units in market | Your true denominator — the impressions you bought | How many were ever seen by someone with a phone |
| Scans | That the code, placement and promise work | Anything about revenue. A curiosity scan and a purchase intent scan are identical here |
| Sessions started | That the destination loaded before people gave up | Whether it loaded fast. Track the gap between scans and sessions — it's your latency tax |
| Orders | The programme produced transactions | Whether those transactions are new revenue |
| Repeat orders | You acquired a relationship, not just a sale | Whether the relationship would have formed anyway |
| Incremental revenue | The only number worth defending in a board meeting | Nothing — but you can only get it with a control |
The gap between the fourth row and the sixth is where most programmes get argued to death internally. It is entirely avoidable, and the cost of avoiding it is one decision made before the print run.
Every measurement problem in this article is cheap to solve in advance and impossible to solve afterwards.
Four designs, ranked
| Design | How it works | Strength | What breaks it |
|---|---|---|---|
| 1. Randomised batch holdout | Split the print run. Half the units carry codes, half don't, distributed at random through the same channels. | The only design that gives a clean causal read. Everything else in the world happened to both cells. | Production complexity, and a supply chain that can't randomise which cases go where. |
| 2. Geographic split | Codes in some regions or store groups, not others. Match cells on baseline sales, not on population. | Usually feasible with real distribution. Good enough for most decisions. | Regions differ in ways you didn't match on. Contamination when customers move between them. |
| 3. Staggered rollout | Roll out across SKUs or regions in waves; each not-yet-live group is a temporary control. | Nobody has to be permanently excluded, which makes it politically easy to approve. | Needs several waves and patience. Weak if the whole market moves during the rollout. |
| 4. Before and after | Compare the period after launch to the period before. | Requires no planning at all. | Everything. Seasonality, promotions, paid spend, distribution changes, weather. This is not evidence; treat it as a sanity check only. |
If you can run design 1, run it. If you can't, run 2. Design 4 is what you end up with when nobody made a decision in time — and it's the reason so many perfectly good programmes get cancelled by a sceptical CFO who is, strictly speaking, right.
Design the control before you print
Practical notes for making this survive contact with a real supply chain:
- Encode the cell in the URL. A path or parameter per batch, region or wave means the analysis is a filter rather than an archaeology project. Do it even if you think you'll only ever run one cell.
- Keep the code destination dynamic. Codes are printed once and live for years. If you can't change where they point, you can't run a second experiment and you can't survive a discontinued SKU.
- Match cells on baseline sales, not headcount. Two regions with equal population and unequal distribution are not a matched pair.
- Decide the window before you start, and make it at least one full purchase cycle for the category. For most CPG reorder behaviour that's 8–12 weeks. Choosing the window after seeing the data is how you accidentally lie to yourself.
- Write down the expected effect size in advance. If the honest answer is "we don't know", pick the smallest effect that would justify the programme, and check your sample is big enough to detect it. A test that can't detect a result you'd act on is a waste of a print run.
The confounders that eat results
Keep a dated log from the day the test starts. Not a memory — a file. Every one of these has ruined an otherwise good read:
- Seasonality and weather, especially for drinks and anything gifted
- Promotional calendar, yours and the retailer's
- Paid media spend and mix, by region if you buy regionally
- Distribution changes — new listings, delistings, out-of-stocks
- Price changes, yours or a competitor's
- PR, reviews, a piece of unexpected virality
- Site or app changes shipped mid-window. This one is on you, and it's the most common
The question finance will ask
Sooner or later: "How much of this would we have sold anyway, through retail, and what did that cost us in the trade relationship?" It's the right question, and having an answer ready is the difference between a programme that scales and one that gets quietly defunded.
You can't measure cannibalisation exactly, but you can bound it. In a geographic or batch design, compare retail sell-through in the exposed and control cells over the same window. If sell-through is flat between them while direct orders rose, the direct orders were largely additive. If exposed-cell sell-through fell, you have a number to subtract, and it's better that you found it than that someone else did.
Illustrative model — no customer data, no Forage data
Inputs, all chosen for the arithmetic: a geographic split (design 2) — 120,000 units into coded regions and 120,000 into matched control regions over 12 weeks · £28 average order value · £11 contribution per direct order · £4 contribution per unit sold through retail · £? programme cost over the window (artwork, print delta, platform fee) — fill in your own. Every figure below is derived from those inputs and the observed counts in the table. This is not a result and not a forecast. It shows the shape of the calculation you should be able to complete.
| Observed over the window | Exposed cell | Control cell | Difference |
|---|---|---|---|
| Scans | 1,800 | — | — |
| Direct orders | 216 | 32 | +184 |
| Direct revenue @ £28 | £6,048 | £896 | +£5,152 |
| Direct contribution @ £11/order | £2,376 | £352 | +£2,024 |
| Retail units sold through | 101,400 | 101,700 | −300 |
| Retail contribution @ £4/unit | £405,600 | £406,800 | −£1,200 |
| Gross contribution effect | +£824 | ||
| Programme cost over the window | your number | −£? | |
Look at what the control did to the story. The headline "£6,048 of direct sales from codes" becomes £5,152 once you subtract the orders the control regions produced anyway, and the honest contribution line is +£2,024 before programme costs.
And now the awkward part, which is the actual lesson
That −300 retail units is not a finding. It is a 0.3% difference between two cells of roughly 100,000 units — comfortably inside the noise a test this size can resolve. Subtracting £1,200 from the result and presenting £824 as "the number" would be exactly the error this article exists to warn about: defending a figure your own design cannot see. The correct read is direct contribution is clearly up £2,024; cannibalisation is unresolved at this sample size, and the test would need far more units per cell — or a much larger true effect — to detect it. Then subtract your programme cost, and say plainly what you still don't know.
Even so, the point worth making to finance survives: on a first 12-week window with no repeat purchases counted yet, the direct contribution is real and measurable. The direct orders also produced 216 identified customers you can reach again at no acquisition cost, and that's where the return actually accumulates. We argued the case for earning the second order without discounting it separately.
The standard we'll hold ourselves to
Since this article exists partly because we don't yet have a customer result to show you, it's fair to state what one will have to contain before we publish it:
- A real, identifiable customer — named, or anonymised in terms they agreed to in writing.
- Numbers from a source they control, not only our dashboard. We're an interested party; our own reporting isn't sufficient evidence for a claim about someone else's revenue.
- The measurement design, stated up front — which of the four above, what the comparison was, what the window was, and what else changed.
- Percentages with their absolute bases. "+34%" means nothing without "from 6.2% to 8.3%, on 3,400 orders".
- A "what we can't claim" section, kept in the final edit.
- Written sign-off from the customer, on file.
If a piece on this site is ever labelled a case study and is missing any of those, we've broken our own rule and you should tell us.
Before you print anything
Write down the decision this test will inform
"Do we roll this out across the range?" is a decision. "Does it work?" isn't, and it can't be answered.
Pick the design
Batch holdout if the supply chain allows, geographic split otherwise. Write down why you rejected the better one.
Encode the cell in the URL
Before the artwork goes to print. This is the step that is genuinely impossible to add later.
Set the window and the minimum effect
One full purchase cycle. The smallest lift that would change the decision. Both written down and dated.
Start the confounder log on day one
Promotions, spend, listings, price, site changes. It takes two minutes a week and it's the difference between a defensible result and an argument.
Agree who computes the number and where it comes from
Ideally the customer's own analytics, ideally someone who isn't selling anything.
Sources
No external statistics are used in this article. The worked example is an illustrative model built from the inputs stated beside it; it contains no customer data and no Forage platform data. Model, not a measurement