Doubled. That’s the number craft retailer Michaels put on the record when describing what its new Google-powered AI shopping assistant did to conversion rates compared with traditional search (Digiday). No cohort definition. No confidence interval. No third party anywhere near the audit trail. Just a headline-ready multiple, offered up in the same breath as a vendor relationship.
That should set off alarms for anyone in publishing who’s currently being pitched an AI tool for subscription acquisition or retention — which, at this point, is most of the industry. The Michaels case, as reported by Digiday, is a clean example of a pattern publishers are going to see a lot more of over the next year: a company adopts a generative AI layer built by or with a platform vendor, then reports a lift metric that happens to make both the retailer and the vendor look good, with no independent verification of how the number was calculated, what the baseline period was, or what “conversion” even meant in context (Digiday).
The Math Question Nobody’s Answering
A “doubled conversion rate” claim is only meaningful once you know what it’s a ratio of. Doubled from what base? Over what time window? Against what control — the same shoppers using old search during the same season, or a different segment entirely? Digiday’s reporting on the Michaels case doesn’t include that methodology, and that’s the point: the retailer’s own statement is the only source of the figure, with the AI vendor’s infrastructure underneath it (Digiday).
Publishers running subscription funnels should recognize the shape of this problem immediately, because it’s structurally identical to the “lift” numbers that show up in vendor case studies for paywall optimization, churn-prediction models, and AI-driven personalization tools. A vendor supplies the AI layer, the publisher runs it against some subset of traffic, and a percentage gets reported — often by the vendor itself, sometimes co-branded with the publisher’s name attached for credibility. The number moves from sales deck to press release to trade coverage without ever passing through an analytics team that wasn’t involved in building the tool.
Why This Matters More for Subscriptions Than for E-Commerce
Retail conversion is a single-touch decision: someone searches, something converts, or it doesn’t. Subscription acquisition and retention are cohort businesses. The number that matters isn’t a topline conversion rate at all — it’s what happens to that cohort’s ARPU, its 90-day churn, its LTV relative to acquisition cost, three and six and twelve months out. An AI-powered paywall tool or churn-prediction model can absolutely lift a short-term signup rate while doing nothing for, or actively hurting, long-run retention if it’s optimizing for the wrong moment in the funnel — say, pushing conversions from lower-intent readers who churn faster once the AI’s personalized offer stops being novel.
That’s a distinction vendor case studies almost never make, because “conversion doubled” is a much better headline than “conversion rose modestly among readers who churned at the same rate as before.” Any publisher evaluating an AI acquisition or retention tool should be asking the vendor, before signing anything, for the denominator behind the topline number, the length of the measurement window, whether the comparison group was concurrent or historical, and — critically — what happened to that cohort’s retention curve after the initial conversion event. If the vendor can’t produce cohort-level data past the first 30 days, the “lift” claim is describing a moment, not a business outcome.
The Audit Gap
What makes the Michaels situation instructive rather than damning is that it’s not really an accusation against Michaels or Google — it’s a description of how self-reported AI performance metrics travel through trade press and into industry consciousness with no verification layer attached (Digiday). Once a number like “doubled conversion” enters circulation, it gets cited by other companies pitching similar tools, cited again by trade publications covering those pitches, and eventually treated as settled fact even though nobody outside the original vendor-client relationship has seen the underlying data.
Publishers have been through this cycle before with programmatic yield claims and with early subscription-platform vendors promising outsized retention gains from recommendation engines. The lesson from that earlier wave holds here: any lift number that comes packaged with a vendor’s product announcement should be treated as a marketing claim until a publisher’s own data team, or an outside auditor, reproduces it against their own baseline. That’s not cynicism — it’s the same rigor publishers already apply to their own churn and ARPU reporting internally. The standard shouldn’t drop just because the tool making the claim has “AI” in its name.
Before adopting any AI system for subscription funnels, revenue teams should insist on the same discipline they’d apply to an internal cohort analysis: raw numerators and denominators, a defined comparison period, and retention data that extends well past the initial conversion event. If a vendor’s case study can’t survive that scrutiny, the “doubled” headline is doing the vendor’s job, not the publisher’s.
