Sample case study
Serum refills fell, and the cause was a pump change nobody had tested
Fernhill BotanicsClean skincare for sensitive skin, sold online31,000 customers · $4.2M revenue · Team of 9
Sample case study. Fernhill Botanics is a fictional brand and the data is simulated to show how Cufo works. These are not customer results.
See this case inside Cufo: open the demo
Refills of Fernhill's Night Repair Serum fell from 34% to 27% in six weeks. The founder first tested an earlier refill reminder, and that test was inconclusive; a separate test showed cheaper shipping was not a lever either. Cufo traced the fall to a cheaper pump shipped in September without a test, and the fix, a tested dispenser, now waits for approval.
- pump complaints in the six weeks after the pump changed
- 116
- serum 60-day refill rate, weeks of 31 Aug to 12 Oct
- 34% to 27%
- pump complaints per serum order since 2 Sep, against 0.3 before
- 9.8 per 100
- customers who bought the serum with the new pump
- 1,180
Situation
The situation
Fernhill Botanics sells four skincare products for sensitive skin online, with a subscription for refills. Night Repair Serum is its second-best seller and its most dependable reorder, at $58 a bottle.
In September the serum's 60-day refill rate began to slip. By the week of 21 Sep it had moved from 34% to 33%, and the founder suspected the refill reminder, sent on day 45, reached people after they had run out. The question that week was simple: why are serum reorders down, and what should change?
The decisions here are the founder’s.
1Data
See: What was collected
Cufo brought six sources together, removed duplicates and personal details, and coded the text into themes. Each record is linked to a customer where the source allows it.
Sources and records
- Product reviews1,912
- Support tickets4,310
- Post-purchase survey answers1,104
- Subscription cancellation reasons212
- The shop's orders48,200
- Site and ad analytics, daily totals40,000
- In all95,738
Gaps Cufo flagged
- 148 of 212 subscription cancellations in 60 days gave no reason, so the view of why subscribers leave is partial.
- The cancellation reasons stopped syncing on 13 Oct, when the connection's access expired.
- 212 support tickets still held order or phone numbers the redaction step had missed, so they were flagged for a second pass.
2Analytics
Think: What it means
Cufo looked for what had changed, when it started, and whether other explanations fit the same weeks. One theme stood out from the rest.
Themes
Each chart shows mentions a week over the 12 weeks before the case, oldest on the left.
Pump leaks or jams
116 mentions
Customers say the serum pump stops halfway, leaks from the side or snaps off. 116 since 2 Sep, against 3 in the six weeks before.
Gentle on sensitive and reactive skin
236 mentions
The most consistent praise, steady for twelve weeks and spread across every product. The formula is not the problem.
Fast, tidy delivery
44 mentions
Shipping is a strength, not a barrier, which fits the result of the shipping test.
In customers’ words
Illustrative lines from the simulated data, with personal details removed.
Third bottle in a row where the pump gives up halfway. I'm decanting it into an old dropper bottle now, which is ridiculous.
Review
hi team, the pump on my sept delivery leaks from the side when I press it. sticky mess on the shelf every day
Support ticket
Still love it. The new bottle feels cheaper though and the pump is hit or miss.
Survey answer
The finding
The cheaper pump that shipped on 2 Sep fails to dispense or cracks in transit, and that is what is costing serum refills.
Evidence
- Fact116 pump complaints on 1,180 serum orders since 2 Sep, or 9.8 per 100, against 3 on about 1,140 orders in the six weeks before.
- FactThe complaints came from three channels at once: 63 support tickets, 41 reviews and 12 survey answers.
- FactBuyers with the new pump refilled less: 702 of 2,600 reordered within 60 days, against 843 of 2,480 with the old pump.
- FactRefund requests that mention the serum rose from 19 to 44 in matched six-week windows.
Confidence
High
Complaints start the week the pump shipped, in three channels at once, while the cleanser, with no packaging change, held its refill rate.
Considered and set aside
- A price change. The serum has cost $58 since March, and mentions of price are flat.
- A seasonal dip across the range. Oat Milk Cleanser refills held over the same weeks: 779 of 1,900 buyers, then 800 of 1,950.
- Reminder timing as the main cause. It was tested, and the test could not tell the two reminder days apart while broken pumps pulled both groups down.
3Experiments
Try: What was tested
Two tests had already run when the pump finding arrived. One checked the founder's first explanation, the other a separate idea about shipping. The fix for the pump is the third.
The tests
-
Test 1 of 3Complete
Serum refill reminder at day 38, not 45
Hypothesis Refills dipped because the reminder lands after people run out.
- Design
- A/B test. People were split at random into two groups, one with the change and one without.
- Measure
- Serum 60-day refill rate. Higher is better.
- Dates
- 22 Sep to 18 Oct 2026
- The rule
- Adopt only if the difference is clear at p below 0.05, in the right direction.
- Rule set in advance
- Yes, before the test started
What the numbers say
No clear difference
29.8% for the control (180 of 604) against 30.7% for the variant (184 of 600), a difference of 0.9 points in the variant's favour; the test could not tell this apart from chance (p = 0.74).
Control: reminder at day 45Variant: reminder at day 38Lines show 95% intervals
Exact figures
Group Total Counted Rate 95% interval Control: reminder at day 45 604 180 29.80% 26.29% to 33.57% Variant: reminder at day 38 600 184 30.67% 27.11% to 34.47% Difference, variant minus control +0.87 points, 95% interval −4.31 to 6.04 (Newcombe) Relative change 2.9% higher p-value, pooled two-proportion z-test 0.7437 (the rule needs below 0.05) Total: people or sessions in each group. Counted: how many of them did what the measure counts. All figures are computed on this page from those two counts.
What the brand did
Inconclusive The founder kept the reminder at day 45. With about 600 people a side, the test could only have caught a change far larger than the one hoped for, and broken pumps were pulling both groups down, so the cause had been misread.
-
Test 2 of 3Complete
Free shipping from $35 instead of $45
Hypothesis Shipping cost is holding back first orders, and a lower threshold pays for itself.
- Design
- A/B test. People were split at random into two groups, one with the change and one without.
- Measure
- Margin per visitor. Higher is better.
- Dates
- 8 Sep to 5 Oct 2026
- The rule
- Adopt only if the difference is clear at p below 0.05, in the right direction.
- Rule set in advance
- Yes, before the test started
What the numbers say
Figures as recorded
This page computes results only from counts of people. This test measured an amount, so its figures are given as recorded, below.
What the brand did
Not adopted Margin per visitor came in at $2.71 with free shipping from $35, against $2.84 at $45, over 106,000 visitors a side. Free shipping stays at $45: only 6 of 1,104 survey answers mention shipping cost, and order size fell with the lower line.
-
Test 3 of 3Planned
New pump on Night Repair Serum
Hypothesis A working pump brings pump complaints from 9.8 per 100 serum orders back toward the 0.3 seen before the September pump.
- Design
- Fix and monitor. The fix goes to everyone and the measure is watched against the period before.
- Measure
- Pump complaints per 100 serum orders. Lower is better.
- Dates
- Planned to run for 31 days once approved
- The rule
- Adopt only if the difference is clear at p below 0.05, in the right direction.
- Rule set in advance
- Yes, written into the plan before the start
What the numbers say
Not run yet
What the brand did
Planned Proposed in the weekly brief of 19 Oct with a free replacement pump for the 1,180 recent buyers, at a cost of $6,490. Refills are read 60 days after the test ends, with a target of 32% or more.
Guardrails
- Serum refund requests per 100 ordersLimit at most 4. Not observed yet.Limit set
4Strategy
Act: What was decided
Every decision goes on the record with what it rested on. The one that caused the problem is the one that was never tested.
Decisions on the record
Tested: decided from a test. Untested: decided without one. Pending: not decided yet.
- Untested
Switch the serum to a cheaper pump from a new supplier.
Shipped without a test. This is the change the complaints point to.
- Tested
Keep free shipping at $45.
Shipping cost came off the list of levers.
- Tested
Keep the refill reminder at day 45.
The test was inconclusive, so the default stayed.
- Pending
Replace the September pump with a tested dispenser and send a free replacement pump to 1,180 recent buyers.
In the weekly brief, waiting for approval before a promotion starts on 23 Oct.
What changed next
- Every test plan now carries a power check before approval.
- The brief now looks at product and message before shipping cost.
- Pump complaints are tracked against the 0.3 per 100 before the September pump, with a target of 1 per 100.
- The refill rate will be read 60 days after the fix, with a target of 32% or more.
Takeaways
What this case shows
The first explanation is not always the right one. A small, fair test kept the brand from building on it.
Reviews, tickets and survey answers read together pointed to a change no one had tested.
An inconclusive test is still worth recording: it showed the cause had been misread and what the next test needs.
See this case inside Cufo: open the demo
More
More sample case studies
-
Sample case study 02
Tidewell Home
Home goods sold online and in 40 stores
A fictional home goods retailer sees fewer first-time buyers order again. Two finished tests, three untested decisions and one open question follow.
17.8%of first-time buyers from April to June ordered again within 90 days. A year earlier: 23.5%
-
Sample case study 03
Brightloop
Language-learning app with AI speaking practice, sold to individuals and schools
A language app checks a price rise it never tested, finds shared logins wiping out progress, and plans a family plan test before launch.
574support tickets in 12 weeks about progress overwritten on a shared login
-
Sample case study 04
Larkfield Mobile
A challenger mobile network in 14 European markets
A fictional mobile network sent an app update to everyone at once. See how Cufo found the cause, stopped a risky test and planned a staged fix.
2.1% to 9.8%port-in eSIM failures, before 9 Oct and at the peak on 12 Oct
Waitlist
Hear when Cufo opens
Cufo is in private development. Early access opens to a small group of brands first.