Join the waitlist

Sample case study

Serum refills fell, and the cause was a pump change nobody had tested

Fernhill BotanicsClean skincare for sensitive skin, sold online31,000 customers · $4.2M revenue · Team of 9

Sample case study. Fernhill Botanics is a fictional brand and the data is simulated to show how Cufo works. These are not customer results.

See this case inside Cufo: open the demo

Refills of Fernhill's Night Repair Serum fell from 34% to 27% in six weeks. The founder first tested an earlier refill reminder, and that test was inconclusive; a separate test showed cheaper shipping was not a lever either. Cufo traced the fall to a cheaper pump shipped in September without a test, and the fix, a tested dispenser, now waits for approval.

pump complaints in the six weeks after the pump changed
116
serum 60-day refill rate, weeks of 31 Aug to 12 Oct
34% to 27%
pump complaints per serum order since 2 Sep, against 0.3 before
9.8 per 100
customers who bought the serum with the new pump
1,180

Situation

The situation

Fernhill Botanics sells four skincare products for sensitive skin online, with a subscription for refills. Night Repair Serum is its second-best seller and its most dependable reorder, at $58 a bottle.

In September the serum's 60-day refill rate began to slip. By the week of 21 Sep it had moved from 34% to 33%, and the founder suspected the refill reminder, sent on day 45, reached people after they had run out. The question that week was simple: why are serum reorders down, and what should change?

The decisions here are the founder’s.

1Data

See: What was collected

Cufo brought six sources together, removed duplicates and personal details, and coded the text into themes. Each record is linked to a customer where the source allows it.

Sources and records

  • Product reviews1,912
  • Support tickets4,310
  • Post-purchase survey answers1,104
  • Subscription cancellation reasons212
  • The shop's orders48,200
  • Site and ad analytics, daily totals40,000
  • In all95,738

Gaps Cufo flagged

  • 148 of 212 subscription cancellations in 60 days gave no reason, so the view of why subscribers leave is partial.
  • The cancellation reasons stopped syncing on 13 Oct, when the connection's access expired.
  • 212 support tickets still held order or phone numbers the redaction step had missed, so they were flagged for a second pass.

2Analytics

Think: What it means

Cufo looked for what had changed, when it started, and whether other explanations fit the same weeks. One theme stood out from the rest.

Themes

Each chart shows mentions a week over the 12 weeks before the case, oldest on the left.

  • Pump leaks or jams

    116 mentions

    Customers say the serum pump stops halfway, leaks from the side or snaps off. 116 since 2 Sep, against 3 in the six weeks before.

    2812 weeks agoLast week
  • Gentle on sensitive and reactive skin

    236 mentions

    The most consistent praise, steady for twelve weeks and spread across every product. The formula is not the problem.

    2112 weeks agoLast week
  • Fast, tidy delivery

    44 mentions

    Shipping is a strength, not a barrier, which fits the result of the shipping test.

    412 weeks agoLast week

In customers’ words

Illustrative lines from the simulated data, with personal details removed.

  • Third bottle in a row where the pump gives up halfway. I'm decanting it into an old dropper bottle now, which is ridiculous.

    Review

  • hi team, the pump on my sept delivery leaks from the side when I press it. sticky mess on the shelf every day

    Support ticket

  • Still love it. The new bottle feels cheaper though and the pump is hit or miss.

    Survey answer

The finding

The cheaper pump that shipped on 2 Sep fails to dispense or cracks in transit, and that is what is costing serum refills.

Evidence

  • Fact116 pump complaints on 1,180 serum orders since 2 Sep, or 9.8 per 100, against 3 on about 1,140 orders in the six weeks before.
  • FactThe complaints came from three channels at once: 63 support tickets, 41 reviews and 12 survey answers.
  • FactBuyers with the new pump refilled less: 702 of 2,600 reordered within 60 days, against 843 of 2,480 with the old pump.
  • FactRefund requests that mention the serum rose from 19 to 44 in matched six-week windows.

Confidence

High

Complaints start the week the pump shipped, in three channels at once, while the cleanser, with no packaging change, held its refill rate.

Considered and set aside

  • A price change. The serum has cost $58 since March, and mentions of price are flat.
  • A seasonal dip across the range. Oat Milk Cleanser refills held over the same weeks: 779 of 1,900 buyers, then 800 of 1,950.
  • Reminder timing as the main cause. It was tested, and the test could not tell the two reminder days apart while broken pumps pulled both groups down.

3Experiments

Try: What was tested

Two tests had already run when the pump finding arrived. One checked the founder's first explanation, the other a separate idea about shipping. The fix for the pump is the third.

The tests

  1. Test 1 of 3Complete

    Serum refill reminder at day 38, not 45

    Hypothesis Refills dipped because the reminder lands after people run out.

    Design
    A/B test. People were split at random into two groups, one with the change and one without.
    Measure
    Serum 60-day refill rate. Higher is better.
    Dates
    22 Sep to 18 Oct 2026
    The rule
    Adopt only if the difference is clear at p below 0.05, in the right direction.
    Rule set in advance
    Yes, before the test started

    What the numbers say

    No clear difference

    29.8% for the control (180 of 604) against 30.7% for the variant (184 of 600), a difference of 0.9 points in the variant's favour; the test could not tell this apart from chance (p = 0.74).

    26%28%30%32%34%36%ControlVariant

    Control: reminder at day 45Variant: reminder at day 38Lines show 95% intervals

    Exact figures
    GroupTotalCountedRate95% interval
    Control: reminder at day 4560418029.80%26.29% to 33.57%
    Variant: reminder at day 3860018430.67%27.11% to 34.47%
    Difference, variant minus control+0.87 points, 95% interval −4.31 to 6.04 (Newcombe)
    Relative change2.9% higher
    p-value, pooled two-proportion z-test0.7437 (the rule needs below 0.05)

    Total: people or sessions in each group. Counted: how many of them did what the measure counts. All figures are computed on this page from those two counts.

    What the brand did

    Inconclusive The founder kept the reminder at day 45. With about 600 people a side, the test could only have caught a change far larger than the one hoped for, and broken pumps were pulling both groups down, so the cause had been misread.

  2. Test 2 of 3Complete

    Free shipping from $35 instead of $45

    Hypothesis Shipping cost is holding back first orders, and a lower threshold pays for itself.

    Design
    A/B test. People were split at random into two groups, one with the change and one without.
    Measure
    Margin per visitor. Higher is better.
    Dates
    8 Sep to 5 Oct 2026
    The rule
    Adopt only if the difference is clear at p below 0.05, in the right direction.
    Rule set in advance
    Yes, before the test started

    What the numbers say

    Figures as recorded

    This page computes results only from counts of people. This test measured an amount, so its figures are given as recorded, below.

    What the brand did

    Not adopted Margin per visitor came in at $2.71 with free shipping from $35, against $2.84 at $45, over 106,000 visitors a side. Free shipping stays at $45: only 6 of 1,104 survey answers mention shipping cost, and order size fell with the lower line.

  3. Test 3 of 3Planned

    New pump on Night Repair Serum

    Hypothesis A working pump brings pump complaints from 9.8 per 100 serum orders back toward the 0.3 seen before the September pump.

    Design
    Fix and monitor. The fix goes to everyone and the measure is watched against the period before.
    Measure
    Pump complaints per 100 serum orders. Lower is better.
    Dates
    Planned to run for 31 days once approved
    The rule
    Adopt only if the difference is clear at p below 0.05, in the right direction.
    Rule set in advance
    Yes, written into the plan before the start

    What the numbers say

    Not run yet

    What the brand did

    Planned Proposed in the weekly brief of 19 Oct with a free replacement pump for the 1,180 recent buyers, at a cost of $6,490. Refills are read 60 days after the test ends, with a target of 32% or more.

    Guardrails

    • Serum refund requests per 100 ordersLimit at most 4. Not observed yet.Limit set

4Strategy

Act: What was decided

Every decision goes on the record with what it rested on. The one that caused the problem is the one that was never tested.

Decisions on the record

Tested: decided from a test. Untested: decided without one. Pending: not decided yet.

  1. Switch the serum to a cheaper pump from a new supplier.

    Shipped without a test. This is the change the complaints point to.

    Untested
  2. Keep free shipping at $45.

    Shipping cost came off the list of levers.

    Tested
  3. Keep the refill reminder at day 45.

    The test was inconclusive, so the default stayed.

    Tested
  4. Replace the September pump with a tested dispenser and send a free replacement pump to 1,180 recent buyers.

    In the weekly brief, waiting for approval before a promotion starts on 23 Oct.

    Pending

What changed next

  • Every test plan now carries a power check before approval.
  • The brief now looks at product and message before shipping cost.
  • Pump complaints are tracked against the 0.3 per 100 before the September pump, with a target of 1 per 100.
  • The refill rate will be read 60 days after the fix, with a target of 32% or more.

Takeaways

What this case shows

  1. The first explanation is not always the right one. A small, fair test kept the brand from building on it.

  2. Reviews, tickets and survey answers read together pointed to a change no one had tested.

  3. An inconclusive test is still worth recording: it showed the cause had been misread and what the next test needs.

See this case inside Cufo: open the demo

More

More sample case studies

  • Sample case study 02

    Tidewell Home

    Home goods sold online and in 40 stores

    A fictional home goods retailer sees fewer first-time buyers order again. Two finished tests, three untested decisions and one open question follow.

    17.8%of first-time buyers from April to June ordered again within 90 days. A year earlier: 23.5%

  • Sample case study 03

    Brightloop

    Language-learning app with AI speaking practice, sold to individuals and schools

    A language app checks a price rise it never tested, finds shared logins wiping out progress, and plans a family plan test before launch.

    574support tickets in 12 weeks about progress overwritten on a shared login

  • Sample case study 04

    Larkfield Mobile

    A challenger mobile network in 14 European markets

    A fictional mobile network sent an app update to everyone at once. See how Cufo found the cause, stopped a risky test and planned a staged fix.

    2.1% to 9.8%port-in eSIM failures, before 9 Oct and at the peak on 12 Oct

All sample case studies

Waitlist

Hear when Cufo opens

Cufo is in private development. Early access opens to a small group of brands first.

Join the waitlist