35K sessions, one AI pipeline, and a 70% drop-off nobody had measured

Context

NayaOne’s platform onboards fintech vendors so that banks can evaluate them. Everyone knew the vendor onboarding flow had friction — customer success spent real hours walking vendors through it — but nobody could say where it failed or how badly, because the answer was buried in tens of thousands of raw product sessions no one had time to watch.

Problem

A manual UX audit of the session corpus was estimated at 50–100 hours of analyst time, which is why it never happened. Meanwhile the anecdotal complaints kept costing customer success hours and the business kept losing vendors silently, before they ever finished signing up.

Constraints

• 35K+ sessions — far past what any human review could cover.
• Session data lived in an analytics store not designed for this question; the pipeline had to extract, normalize, and classify it without disturbing production.
• LLM classification at this volume had to be cheap, repeatable, and skeptically validated — an analysis nobody trusts changes nothing.

My role

I built the analysis system end-to-end — extraction, the PostgreSQL processing layer, and LLM-driven classification via the Gemini API — then took the findings to the CEO as a business case — which put the redesign on the quarterly roadmap — and led the five-person team that implemented it.

Approach

Sessions don’t exist in raw analytics data — only events do. The pipeline reconstructed them inside PostgreSQL with window functions over the event stream, using a 30-minute inactivity gap as the session boundary, then reduced each session to a structured event sequence with per-step timings. User identifiers were anonymized to opaque IDs before anything left the database. The LLM layer then classified where in the onboarding funnel each session stalled and what the user was attempting at that moment, in two tiers: a per-tenant analysis pass, and a cross-tenant synthesis that rolled the per-tenant findings into one platform-level picture. Aggregating the classifications turned tens of thousands of individual recordings into a funnel with numbers on it.

The first full run took about 1.5 hours and roughly $7 in model usage — against the 50–100 hour manual estimate — and produced the headline nobody had: 70% of vendors abandoned onboarding before completing it, concentrated at a small number of identifiable steps.

Key decisions

• Classify sessions, don’t summarize them. A fixed taxonomy of funnel stages and failure modes made the output aggregable and auditable; free-form LLM summaries would have been unverifiable prose.
• Distinguish friction from engagement. Long dwell time on a data view is a success signal; long dwell time on a form field is a failure signal. The classification taxonomy had to encode that difference explicitly, or every engaged user would have read as a stuck one and the funnel numbers would lie.
• Spot-check against ground truth. A sample of classifications was verified against the actual recordings before anyone made decisions on the numbers — the model earned trust before it spent it.
• Lead the fix, not just the finding. Analysis that ends at a slide deck is a cost. I took the redesign through my team as the follow-through.

Trade-offs

A fixed classification taxonomy misses failure modes you didn’t anticipate — some nuance in those sessions never made it into a category. That was the right price for numbers that could be compared, re-run, and defended in front of stakeholders.

Result

The redesign cut vendor onboarding abandonment from 70% to under 5%, and the self-service flow eliminated roughly 200–300 customer success hours per year. The pipeline itself became reusable: the same 1.5-hour analysis could be re-run after any product change to verify the funnel held.

What I’d do differently

Instrument the funnel with proper analytics events from day one, so the question “where do users give up?” is a dashboard, not a data-engineering project. The AI pipeline was a brilliant retrofit — but it was a retrofit.