The AI feature that cut churn, and the one that didn't

In short: on a platform product I led AI on, a round of AI features came with lower churn and higher feature adoption. Those are two different measures, and a feature can win one without touching the other. If you want to know which of your AI features keeps people, measure retention by cohort, not usage.

On one platform product, the AI features were my job. The work was the unglamorous kind: AI pipelines and data infrastructure, keeping LLM calls fast and reliable in production, cloud services, and adding new AI capabilities to the product one at a time. Over that stretch, two numbers moved in the right direction.

It is tempting to put those side by side and say the AI features cut churn. I want to be more careful than that, because the two numbers measure different things, and confusing them is how teams end up shipping features people use and still leave.

Used is not the same as kept

Adoption asks: of the people who could use this feature, how many did? Churn asks: of the people who were here, how many left? A feature can be tried by everyone in its first week and have no effect on whether anyone stays. Novelty drives adoption. Only usefulness drives retention.

That is the honest version of this post's title. Both movements are for a whole period and a set of features, not for one feature each. I am not going to pretend I can split a churn number cleanly across features that shipped close together. What I can do is tell you how I think about which kind of AI feature tends to move each number, and how to check it on your own product.

Two kinds of AI feature

The test I like is the last row. If a feature broke for a day, would anyone write in? If not, it is probably not the reason anyone stays.

What the curves should look like

The way to see this is a retention curve split by whether a user picked up the feature early. Here is the shape to look for. This drawing is illustrative, not data from the product.

If both curves fall together, the feature is being used and is not keeping anyone. That is the feature that did not cut churn, and every product has a few.

How to measure it

You need two tables you almost certainly have already: users with a sign-up date, and events with a user, a name and a timestamp. This query splits a cohort by early use of one feature and counts who is still active each week.

WITH cohort AS (
  SELECT u.id, u.created_at,
         EXISTS (
           SELECT 1 FROM events e
           WHERE e.user_id = u.id
             AND e.name = 'ai_feature_used'
             AND e.at < u.created_at + interval '14 days'
         ) AS adopted
  FROM users u
  WHERE u.created_at < now() - interval '70 days'
),
active AS (
  SELECT DISTINCT e.user_id,
         floor(extract(epoch FROM e.at - c.created_at) / 604800)::int AS week
  FROM events e
  JOIN cohort c ON c.id = e.user_id
)
SELECT c.adopted, a.week,
       round(count(DISTINCT a.user_id)::numeric
             / (SELECT count(*) FROM cohort c2 WHERE c2.adopted = c.adopted), 3) AS retained
FROM cohort c
JOIN active a ON a.user_id = c.id
WHERE a.week BETWEEN 0 AND 10
GROUP BY c.adopted, a.week
ORDER BY c.adopted, a.week;

One warning about reading it. People who try a new feature early are often your most engaged users anyway. They would have stayed without it. So a gap between the curves is a hint, not a cause. If you can, hold the feature back from a random slice of new users for a few weeks and compare that slice instead. That is the only version of this I would put in front of a board.

What to do on Monday

1. For each AI feature, write down which number you expect it to move: adoption, retention, or both. Most teams have never written this down.
2. Log one clear event per feature, with a stable name. You cannot measure what you did not log.
3. Run the cohort query for your two most-used AI features. Look for a gap that stays open after the first few weeks.
4. For the feature with no gap, ask the "if it broke for a day" question. Then decide whether to move it closer to the task people already do, or stop investing in it.
5. Next time you ship, hold it back from a random slice of users first. It costs a few weeks and saves you from arguing about correlation later.