Join the waitlist

Sample case study

An app update broke eSIM activation for people moving their number

Larkfield MobileA challenger mobile network in 14 European markets2,100,000 customers · €410M revenue · 8 teams across 14 markets

Sample case study. Larkfield Mobile is a fictional brand and the data is simulated to show how Cufo works. These are not customer results.

Larkfield Mobile sent an app update to every customer at once, and eSIM activation broke for people moving their number. Cufo flagged the problem the next day and the activation logs confirmed the cause. A save-offer call test was stopped when its complaints guardrail broke, the fix is planned in stages and waits for approval, and a referral test is still a draft.

port-in eSIM failures, before 9 Oct and at the peak on 12 Oct
2.1% to 9.8%
eSIM activation complaints in 12 weeks
20,200
customers hit by a failed activation since 9 Oct
6,120
of month-2 leavers had a first-week activation, SIM swap or port problem, against 9% of those who stayed
46%

Situation

The situation

Larkfield Mobile is a challenger mobile network with 2,100,000 customers in 14 European markets. This case follows the week of 19 October 2026.

On 9 October, app version 8.2 went to every customer at once. It cut the eSIM download timeout from 300 to 90 seconds, as a faster setup. Soon after, people moving their number to Larkfield could not get a working eSIM. The VP Customer Experience asked one question: how fast can we get port-in eSIM failures under 3%? Other problems ran at the same time: a price rise letter, roaming charges, indoor call drops and long waits for business customers.

The decisions here are the vp customer experience’s.

1Data

See: What was collected

Cufo read seven sources, removed empty and repeated records, masked personal data and tied each record to a theme. Activation logs show every eSIM attempt with its number port status at the time.

Sources and records

  • Web and in-app chat131,000
  • Call transcripts98,000
  • App reviews74,000
  • Survey comments (NPS)72,000
  • Public social posts61,000
  • Cancellation reasons50,000
  • Activation logs31,400
  • In all517,400

Gaps Cufo flagged

  • Spain survey scores before and after 1 Oct are not directly comparable, because invitations moved from email to SMS.
  • 3,420 cancellations use a retired reason code, "Technical problem". It is not yet mapped to the new activation and network codes.
  • 2,900 public posts come from accounts flagged as automated. They are not yet excluded.
  • 6,200 call transcripts still hold spoken phone numbers or bank account numbers that the masking missed.

2Analytics

Think: What it means

Cufo lined up six themes against release dates and network changes. Only eSIM activation moved with an app release. A weaker pattern also showed in cancellations, and it is correlation only.

Themes

Each chart shows mentions a week over the 12 weeks before the case, oldest on the left.

  • eSIM activation fails after number port-in

    20,200 mentions

    People move their number, the eSIM will not download, and they are left with no line.

    492512 weeks agoLast week
  • Confusion about the mid-contract price rise

    27,900 mentions

    People do not understand the letter. It did not say the amount in euros.

    448812 weeks agoLast week
  • Calls drop indoors on 5G in Madrid and Barcelona

    15,200 mentions

    Calls cut out inside homes and offices. Outside, they are fine.

    232212 weeks agoLast week
  • Weekend unlimited data drives word of mouth

    11,300 mentions

    Customers praise the weekend data and say they tell friends. It is the one theme with praise.

    109712 weeks agoLast week

In customers’ words

Illustrative lines from the simulated data, with personal details removed.

  • Two days without a line. The app says "activation failed, try again" and my old number is already gone from my previous operator. I am paying for nothing.

    App review

  • Agent: I can see three failed download attempts, all at 90 seconds. Customer: So is it my phone? Agent: No, it looks like it's on our side.

    Call transcript

  • Activation was a nightmare. The third chat agent finally told me to wait for the port and try again later. That should be on the screen.

    Survey comment

The finding

App v8.2 cut the eSIM download timeout to 90 seconds, so activations stop while the number port is still pending.

Evidence

  • FactPort-in eSIM activations failed 2.1% of the time before 9 Oct and 9.8% on 12 Oct.
  • Fact81% of failed activations had a port-in still pending when they failed, against 34% of successful ones.
  • FactActivations without a port-in stayed flat at 1.4%, so the eSIM service itself is not the problem.
  • FactSpain's survey score fell from 27 to 16 in 12 weeks. For customers under 60 days' tenure it fell from 34 to 9. Treat the size of the drop with care, because the survey invitations changed on 1 Oct.

Confidence

High

Three independent sources agree, and failures began within 6 hours of the release in every market.

Considered and set aside

  • A fault in one phone brand's software. That brand makes up 58% of failures and 56% of port-in eSIM activations, so it is not over-represented.
  • Slower number release by other operators in Spain. The median port time went from 3.1 to 3.4 hours, and failures also rose in Portugal and Italy, where port times are flat. It explains 10% to 15% of the extra failures at most, so it is a small part, not the cause.

3Experiments

Try: What was tested

Larkfield had one finished test, one planned test and one draft. The finished test did not go to plan, and it is shown in full.

The tests

  1. Test 1 of 3Complete

    Save-offer call with a €10 credit, 10% holdout

    Hypothesis A retention call with a €10 credit, within a day of churn-intent language, cuts 60-day churn among customers under 90 days.

    Design
    Holdout. A random group was kept back from the change, to compare against everyone else.
    Measure
    60-day churn after churn-intent contact. Lower is better.
    Dates
    31 Aug to 27 Sep 2026
    The rule
    Adopt only if the difference is clear at p below 0.05, in the right direction.
    Rule set in advance
    Yes, before the test started

    What the numbers say

    No clear difference

    21.4% for the control (108 of 504) against 20.2% for the variant (915 of 4,536), a difference of 1.3 points in the variant's favour; the test could not tell this apart from chance (p = 0.51).

    18%20%22%24%26%ControlVariant

    Control: no outbound call (10% holdout)Variant: retention call with a €10 creditLines show 95% intervals

    Exact figures
    GroupTotalCountedRate95% interval
    Control: no outbound call (10% holdout)50410821.43%18.07% to 25.22%
    Variant: retention call with a €10 credit4,53691520.17%19.03% to 21.36%
    Difference, variant minus control−1.26 points, 95% interval −5.22 to 2.31 (Newcombe)
    Relative change5.9% lower
    p-value, pooled two-proportion z-test0.5058 (the rule needs below 0.05)

    Total: people or sessions in each group. Counted: how many of them did what the measure counts. All figures are computed on this page from those two counts.

    What the brand did

    Stopped The guardrail broke, so the playbook was paused on 1 Oct, before it reached everyone. The rule that asks a person to approve offers of money stays locked on.

    Guardrails

    • Complaints about unwanted calls per 1,000 calledLimit at most 15. Observed about 21.Broken
  2. Test 2 of 3Planned

    Activation flow v8.3, staged rollout

    Hypothesis The v8.3 flow, which waits for the number port before downloading the eSIM profile, brings the port-in failure rate from 8.4% back under 3%.

    Design
    Staged rollout. The change reaches more people step by step, with a check at each step.
    Measure
    Port-in eSIM activation failure rate. Lower is better.
    Dates
    Planned: 10% of port-in activations for 4 days, then 50% for 5 days
    The rule
    Adopt only if the difference is clear at p below 0.05, in the right direction.
    Rule set in advance
    Not yet registered

    What the numbers say

    Not run yet

    What the brand did

    Planned On 19 Oct this is a recommendation waiting for approval. v8.3 has not reached anyone yet. It goes to everyone only if failures fall and both guardrails hold: app crashes during activation at most 0.8%, and activation contacts at most 120 per 1,000 activations.

  3. Test 3 of 3Planned

    Referral nudge after the second unlimited weekend, 10% holdout

    Hypothesis A referral card after the second unlimited weekend lifts referrals among weekend users by 15% or more.

    Design
    Holdout. A random group was kept back from the change, to compare against everyone else.
    Measure
    Referral within 30 days, weekend users. Higher is better.
    Dates
    Not started. Planned to run for 42 days
    The rule
    Adopt only if the chance that the variant is better reaches 95%.
    Rule set in advance
    Not yet registered

    What the numbers say

    Not run yet

    What the brand did

    Planned Still a draft, with no start date and no registration. Today's referral gap, 4.1% against 2.6% for customers who used weekend unlimited twice, is probably part selection, because heavy users refer anyway. A holdout is how Cufo would tell. The guardrail is opt-outs from marketing messages at most 0.6%.

4Strategy

Act: What was decided

Some decisions rested on a test, some did not, and one is still waiting. The page keeps them apart.

Decisions on the record

Tested: decided from a test. Untested: decided without one. Pending: not decided yet.

  1. Make weekend unlimited data standard on the Unlimited 5G plan.

    It was cheap and reversible, so it was adopted directly with a check-back on referrals.

    Untested
  2. Pause the save-offer call playbook.

    The complaints guardrail broke in the holdout test. A text-first offer is being drafted instead of a call.

    Tested
  3. Send an explainer email with the euro amount first to 1,240,000 postpaid customers.

    It was required notice. Price rise mentions fell from 3,031 to 1,266 a week. The email predicted a fall of 40% to 60%, and the measured fall of 58% sits inside that range. With no holdout, Cufo cannot split the email from the usual fade after a first bill.

    Untested
  4. Roll back the 3.5 GHz handover settings on 47 sites.

    Dropped calls on those sites rose from 0.6% to 1.4% after 22 Sep. The rollback covers 18 of 47 sites so far. The check-back is set for 26 Oct.

    Untested
  5. Recommended: release v8.3 in stages, 10%, then 50%, then everyone, and send the apology after that.

    Waiting for approval. The apology text and €10 credit for 6,120 customers, €61,200 in all, is a separate approval. It waits until v8.3 reaches everyone and failures stay under 3% for 7 days.

    Pending

What changed next

  • The v8.3 fix is built and planned to go out in stages. Nothing has shipped yet.
  • The apology to the 6,120 affected customers waits for the fix, then needs an approval.
  • The save-offer call is replaced by a text-first offer. It is still a draft.
  • The referral test is still a draft with no start date.
  • Still open: do customers who join on the half-price offer leave more once activation problems are removed? A survey of month-2 leavers is in draft.
  • Still open: new arrival text for roaming countries is in legal review, and business waits have no owner yet.

Takeaways

What this case shows

  1. A release that reaches everyone at once can break things for everyone. Stages and guardrails limit the damage.

  2. A guardrail that stops a test is the system working, not a failure.

  3. Keeping found, tested, decided and still open apart means a recommendation is never read as a result.

More

More sample case studies

  • Sample case study 01

    Fernhill Botanics

    Clean skincare for sensitive skin, sold online

    Refills of a best-selling serum fell. A reminder test was inconclusive; reviews and tickets pointed to a new pump that leaks or jams.

    116pump complaints in the six weeks after the pump changed

  • Sample case study 02

    Tidewell Home

    Home goods sold online and in 40 stores

    A fictional home goods retailer sees fewer first-time buyers order again. Two finished tests, three untested decisions and one open question follow.

    17.8%of first-time buyers from April to June ordered again within 90 days. A year earlier: 23.5%

  • Sample case study 03

    Brightloop

    Language-learning app with AI speaking practice, sold to individuals and schools

    A language app checks a price rise it never tested, finds shared logins wiping out progress, and plans a family plan test before launch.

    574support tickets in 12 weeks about progress overwritten on a shared login

All sample case studies

The demo shows the Fernhill Botanics case inside Cufo, with the same simulated data: open the demo.

Waitlist

Hear when Cufo opens

Cufo is in private development. Early access opens to a small group of brands first.

Join the waitlist