Sample case study
An app update broke eSIM activation for people moving their number
Larkfield MobileA challenger mobile network in 14 European markets2,100,000 customers · €410M revenue · 8 teams across 14 markets
Sample case study. Larkfield Mobile is a fictional brand and the data is simulated to show how Cufo works. These are not customer results.
Larkfield Mobile sent an app update to every customer at once, and eSIM activation broke for people moving their number. Cufo flagged the problem the next day and the activation logs confirmed the cause. A save-offer call test was stopped when its complaints guardrail broke, the fix is planned in stages and waits for approval, and a referral test is still a draft.
- port-in eSIM failures, before 9 Oct and at the peak on 12 Oct
- 2.1% to 9.8%
- eSIM activation complaints in 12 weeks
- 20,200
- customers hit by a failed activation since 9 Oct
- 6,120
- of month-2 leavers had a first-week activation, SIM swap or port problem, against 9% of those who stayed
- 46%
Situation
The situation
Larkfield Mobile is a challenger mobile network with 2,100,000 customers in 14 European markets. This case follows the week of 19 October 2026.
On 9 October, app version 8.2 went to every customer at once. It cut the eSIM download timeout from 300 to 90 seconds, as a faster setup. Soon after, people moving their number to Larkfield could not get a working eSIM. The VP Customer Experience asked one question: how fast can we get port-in eSIM failures under 3%? Other problems ran at the same time: a price rise letter, roaming charges, indoor call drops and long waits for business customers.
The decisions here are the vp customer experience’s.
1Data
See: What was collected
Cufo read seven sources, removed empty and repeated records, masked personal data and tied each record to a theme. Activation logs show every eSIM attempt with its number port status at the time.
Sources and records
- Web and in-app chat131,000
- Call transcripts98,000
- App reviews74,000
- Survey comments (NPS)72,000
- Public social posts61,000
- Cancellation reasons50,000
- Activation logs31,400
- In all517,400
Gaps Cufo flagged
- Spain survey scores before and after 1 Oct are not directly comparable, because invitations moved from email to SMS.
- 3,420 cancellations use a retired reason code, "Technical problem". It is not yet mapped to the new activation and network codes.
- 2,900 public posts come from accounts flagged as automated. They are not yet excluded.
- 6,200 call transcripts still hold spoken phone numbers or bank account numbers that the masking missed.
2Analytics
Think: What it means
Cufo lined up six themes against release dates and network changes. Only eSIM activation moved with an app release. A weaker pattern also showed in cancellations, and it is correlation only.
Themes
Each chart shows mentions a week over the 12 weeks before the case, oldest on the left.
eSIM activation fails after number port-in
20,200 mentions
People move their number, the eSIM will not download, and they are left with no line.
Confusion about the mid-contract price rise
27,900 mentions
People do not understand the letter. It did not say the amount in euros.
Calls drop indoors on 5G in Madrid and Barcelona
15,200 mentions
Calls cut out inside homes and offices. Outside, they are fine.
Weekend unlimited data drives word of mouth
11,300 mentions
Customers praise the weekend data and say they tell friends. It is the one theme with praise.
In customers’ words
Illustrative lines from the simulated data, with personal details removed.
Two days without a line. The app says "activation failed, try again" and my old number is already gone from my previous operator. I am paying for nothing.
App review
Agent: I can see three failed download attempts, all at 90 seconds. Customer: So is it my phone? Agent: No, it looks like it's on our side.
Call transcript
Activation was a nightmare. The third chat agent finally told me to wait for the port and try again later. That should be on the screen.
Survey comment
The finding
App v8.2 cut the eSIM download timeout to 90 seconds, so activations stop while the number port is still pending.
Evidence
- FactPort-in eSIM activations failed 2.1% of the time before 9 Oct and 9.8% on 12 Oct.
- Fact81% of failed activations had a port-in still pending when they failed, against 34% of successful ones.
- FactActivations without a port-in stayed flat at 1.4%, so the eSIM service itself is not the problem.
- FactSpain's survey score fell from 27 to 16 in 12 weeks. For customers under 60 days' tenure it fell from 34 to 9. Treat the size of the drop with care, because the survey invitations changed on 1 Oct.
Confidence
High
Three independent sources agree, and failures began within 6 hours of the release in every market.
Considered and set aside
- A fault in one phone brand's software. That brand makes up 58% of failures and 56% of port-in eSIM activations, so it is not over-represented.
- Slower number release by other operators in Spain. The median port time went from 3.1 to 3.4 hours, and failures also rose in Portugal and Italy, where port times are flat. It explains 10% to 15% of the extra failures at most, so it is a small part, not the cause.
3Experiments
Try: What was tested
Larkfield had one finished test, one planned test and one draft. The finished test did not go to plan, and it is shown in full.
The tests
-
Test 1 of 3Complete
Save-offer call with a €10 credit, 10% holdout
Hypothesis A retention call with a €10 credit, within a day of churn-intent language, cuts 60-day churn among customers under 90 days.
- Design
- Holdout. A random group was kept back from the change, to compare against everyone else.
- Measure
- 60-day churn after churn-intent contact. Lower is better.
- Dates
- 31 Aug to 27 Sep 2026
- The rule
- Adopt only if the difference is clear at p below 0.05, in the right direction.
- Rule set in advance
- Yes, before the test started
What the numbers say
No clear difference
21.4% for the control (108 of 504) against 20.2% for the variant (915 of 4,536), a difference of 1.3 points in the variant's favour; the test could not tell this apart from chance (p = 0.51).
Control: no outbound call (10% holdout)Variant: retention call with a €10 creditLines show 95% intervals
Exact figures
Group Total Counted Rate 95% interval Control: no outbound call (10% holdout) 504 108 21.43% 18.07% to 25.22% Variant: retention call with a €10 credit 4,536 915 20.17% 19.03% to 21.36% Difference, variant minus control −1.26 points, 95% interval −5.22 to 2.31 (Newcombe) Relative change 5.9% lower p-value, pooled two-proportion z-test 0.5058 (the rule needs below 0.05) Total: people or sessions in each group. Counted: how many of them did what the measure counts. All figures are computed on this page from those two counts.
What the brand did
Stopped The guardrail broke, so the playbook was paused on 1 Oct, before it reached everyone. The rule that asks a person to approve offers of money stays locked on.
Guardrails
- Complaints about unwanted calls per 1,000 calledLimit at most 15. Observed about 21.Broken
-
Test 2 of 3Planned
Activation flow v8.3, staged rollout
Hypothesis The v8.3 flow, which waits for the number port before downloading the eSIM profile, brings the port-in failure rate from 8.4% back under 3%.
- Design
- Staged rollout. The change reaches more people step by step, with a check at each step.
- Measure
- Port-in eSIM activation failure rate. Lower is better.
- Dates
- Planned: 10% of port-in activations for 4 days, then 50% for 5 days
- The rule
- Adopt only if the difference is clear at p below 0.05, in the right direction.
- Rule set in advance
- Not yet registered
What the numbers say
Not run yet
What the brand did
Planned On 19 Oct this is a recommendation waiting for approval. v8.3 has not reached anyone yet. It goes to everyone only if failures fall and both guardrails hold: app crashes during activation at most 0.8%, and activation contacts at most 120 per 1,000 activations.
-
Test 3 of 3Planned
Referral nudge after the second unlimited weekend, 10% holdout
Hypothesis A referral card after the second unlimited weekend lifts referrals among weekend users by 15% or more.
- Design
- Holdout. A random group was kept back from the change, to compare against everyone else.
- Measure
- Referral within 30 days, weekend users. Higher is better.
- Dates
- Not started. Planned to run for 42 days
- The rule
- Adopt only if the chance that the variant is better reaches 95%.
- Rule set in advance
- Not yet registered
What the numbers say
Not run yet
What the brand did
Planned Still a draft, with no start date and no registration. Today's referral gap, 4.1% against 2.6% for customers who used weekend unlimited twice, is probably part selection, because heavy users refer anyway. A holdout is how Cufo would tell. The guardrail is opt-outs from marketing messages at most 0.6%.
4Strategy
Act: What was decided
Some decisions rested on a test, some did not, and one is still waiting. The page keeps them apart.
Decisions on the record
Tested: decided from a test. Untested: decided without one. Pending: not decided yet.
- Untested
Make weekend unlimited data standard on the Unlimited 5G plan.
It was cheap and reversible, so it was adopted directly with a check-back on referrals.
- Tested
Pause the save-offer call playbook.
The complaints guardrail broke in the holdout test. A text-first offer is being drafted instead of a call.
- Untested
Send an explainer email with the euro amount first to 1,240,000 postpaid customers.
It was required notice. Price rise mentions fell from 3,031 to 1,266 a week. The email predicted a fall of 40% to 60%, and the measured fall of 58% sits inside that range. With no holdout, Cufo cannot split the email from the usual fade after a first bill.
- Untested
Roll back the 3.5 GHz handover settings on 47 sites.
Dropped calls on those sites rose from 0.6% to 1.4% after 22 Sep. The rollback covers 18 of 47 sites so far. The check-back is set for 26 Oct.
- Pending
Recommended: release v8.3 in stages, 10%, then 50%, then everyone, and send the apology after that.
Waiting for approval. The apology text and €10 credit for 6,120 customers, €61,200 in all, is a separate approval. It waits until v8.3 reaches everyone and failures stay under 3% for 7 days.
What changed next
- The v8.3 fix is built and planned to go out in stages. Nothing has shipped yet.
- The apology to the 6,120 affected customers waits for the fix, then needs an approval.
- The save-offer call is replaced by a text-first offer. It is still a draft.
- The referral test is still a draft with no start date.
- Still open: do customers who join on the half-price offer leave more once activation problems are removed? A survey of month-2 leavers is in draft.
- Still open: new arrival text for roaming countries is in legal review, and business waits have no owner yet.
Takeaways
What this case shows
A release that reaches everyone at once can break things for everyone. Stages and guardrails limit the damage.
A guardrail that stops a test is the system working, not a failure.
Keeping found, tested, decided and still open apart means a recommendation is never read as a result.
More
More sample case studies
-
Sample case study 01
Fernhill Botanics
Clean skincare for sensitive skin, sold online
Refills of a best-selling serum fell. A reminder test was inconclusive; reviews and tickets pointed to a new pump that leaks or jams.
116pump complaints in the six weeks after the pump changed
-
Sample case study 02
Tidewell Home
Home goods sold online and in 40 stores
A fictional home goods retailer sees fewer first-time buyers order again. Two finished tests, three untested decisions and one open question follow.
17.8%of first-time buyers from April to June ordered again within 90 days. A year earlier: 23.5%
-
Sample case study 03
Brightloop
Language-learning app with AI speaking practice, sold to individuals and schools
A language app checks a price rise it never tested, finds shared logins wiping out progress, and plans a family plan test before launch.
574support tickets in 12 weeks about progress overwritten on a shared login
The demo shows the Fernhill Botanics case inside Cufo, with the same simulated data: open the demo.
Waitlist
Hear when Cufo opens
Cufo is in private development. Early access opens to a small group of brands first.