Long story short
From 5,000 payment records to a decision engine analysts can audit
01 Discovery
Started from the data
Analyzed a 5,000-record payments dataset to find the most common exception patterns, and turned them into a PRD and 5 core decision scenarios.
02 Build
Built the decision engine
Owned the product from scope to working MVP. Mapped user workflows with design, defined the system architecture and built the backend decision pipeline.
03 Evaluation
Tested it against an LLM-only version
Built an evaluation harness for the hybrid design: 100% spec compliance on known patterns, at 91% lower LLM cost.
Core question
How might we help payment analysts resolve suspense payments faster, while keeping every decision explainable and under human control?
Users
The people around every payment
Focus user
Why we started with the Analyst
01 Most volume
They touch all payments
Speeding up their step speeds up every payment.
02 Everything else depends on them
Every escalation, backlog number and report starts with an analyst decision
If their decision is late or unclear, the Investigator and both managers inherit the problem.
03 Measurable Cost
Each case takes 15–20 minutes
Every minute saved repeats across the whole queue.
Pain points
What made every review slow and hard to defend
Pain point 01
Every case starts from zero
Nothing explains why a payment landed in suspense. Analysts spend 15–20 minutes per case querying systems and piecing together enough context to act.
Pain point 02
Every payment breaks differently
A missing reference, a mismatched name, a variable amount, or a duplicate. Because the errors vary wildly, the manual workflow does too.
Pain point 03
Easy and hard cases share one queue
Simple cases are mixed with complicated ones. There's no way to clear the simple cases first when the suspense queue spikes.
Pain point 04
No auditable trail of reasoning
When a decision is audited or handed off to an Investigator, the logic doesn't carry over. The analyst has to explain the case entirely from memory.
Discovery from data
From 5,000 records to 5 scenarios
Without direct access to user observation, we let the data define the problem space. By analyzing a 5,000-record dataset from CES, we grouped exceptions by their failure modes and identified five core patterns behind the queue.
#
Problem
What goes wrong
1
Name mismatch
The policy reference and amount are valid, but the sender's name only partly matches the policyholder ("J Smith" vs "John Smith").
2
Missing policy number
The sender matches a known customer, but the payment has no usable policy reference, and they may hold more than one policy.
3
Amount variance
The customer and policy check out, but the amount differs from the expected premium.
4
Unknown payer
No customer record matches the sender. Sometimes a third party paid on the policyholder's behalf.
5
Duplicate payment
A payment with identical details arrived within 72 hours.
The initial brief offered ideas, not rules. To build the engine, I mapped the five patterns into strict routing scenarios and translated analyst questions into 20 measurable signals, including risk checks that can override a strong match.
Opportunities for the MVP
Turning pain points into a focused first release
Opportunity 01
Pre-assembled Context
The system runs the necessary cross-checks before the analyst opens the case, delivering a recommendation and the underlying evidence in a single view.
Opportunity 02
Standardized Scenario Routing
Instead of a generic manual workflow, every exception is automatically classified into one of five patterns and evaluated by rules specific to that failure mode.
Opportunity 03
Filterable Triage
Analysts can slice their queue by recommendation, scenario, confidence band, or payment method, and sort by confidence or amount, so they can clear the simplest cases first during volume spikes.
Opportunity 04
Traceable Decisions
Every system action is logged, so auditors and investigators inherit the complete story without relying on the analyst's memory.
Key decisions
The calls that shaped PayWise
Each opportunity could be built more than one way. These three choices defined what PayWise does, and what it deliberately doesn't.
01Human approval for every payment
We chose
Explicit analyst approval for every payment, including clear, high-confidence matches.
Instead of
Auto-posting payments that score above a 0.90 confidence threshold.
Why
- A misrouted "Apply" can alter a customer's insurance coverage and creates complex financial reversals.
- Financial accountability requires a designated individual to own money-moving decisions.
- An MVP must establish a proven operational baseline before earning full system autonomy.
Trade-off
The tool is built to speed up each review, but it doesn't shrink the queue on its own.
02Actions over raw scores
We chose
One of three actions (Apply, Hold, or Escalate), with the confidence score shown as evidence.
Instead of
A raw confidence score for the analyst to interpret.
Why
- In the CES dataset, every exception ended in one of these three actions, which matches how analysts already think.
- A score alone lacks context: a 98% match with a risk flag still needs a Hold.
- Categorical recommendations speed up triage, while the score and signals stay visible for verification.
Trade-off
We gave up visibility into effort. Analysts must open a Hold case to see whether it needs a quick look or a deep investigation.
03Every action leaves a traceable record
We chose
A full audit trail for every action.
Instead of
Logging only the final payment status and discarding the intermediate evaluation logic.
Why
- Escalated cases hand over a complete history and a one-page summary report, so investigators don't start from a blank slate.
- Required notes tell us why an analyst rejected a recommendation, which is exactly how we know when to update the rules.
- Anyone can trace a resolved payment back to the data it saw, the confidence score, and the person who made the final call.
Trade-off
We traded speed for system learning. Writing an override reason takes extra time, but an unexplained override provides no data to improve the rules.
The bigger decision
Let code decide, let AI explain
PayWise started LLM-first: a model read each payment, chose the outcome, and explained why. But when I looked closely at the data driving those decisions (match scores, amount variances, yes/no flags), I saw an architectural mismatch. Applying a probabilistic model to numerical thresholds is risky and expensive.
To test that, I gave the LLM every advantage: our full rulebook, the same signals, and step-by-step reasoning. It still changed answers across identical runs, invented constraints we never wrote, and mishandled edge cases.
So I redrew the boundary: code makes the call, and the LLM explains it. Analysts get decisions they can check and reasons in plain language.
Correct recommendations
100%+6 PTS
Consistent decisions
100%ZERO DRIFT
Lower LLM cost
91%11× CHEAPER
per 1,000 payments
Even on payment formats outside the spec, it made zero wrong Applies. Anything it couldn't resolve went to Hold or Escalate.
Product walkthrough
One queue, four roles
Follow exceptions from the analyst's queue to an approved rule change.
Learnings
What I'd carry forward
01Align the architecture to the data
Numbers belong in code; language belongs in an LLM. Recognizing this was a rules problem earlier would have accelerated the build.
02Design for graceful failure
The system already fails safe, with zero wrong Applies on unknown formats. Next, I'd add proactive alerts to flag unreadable payments before they bottleneck the queue.
03Measure trust, not just outcomes
The CES dataset showed what analysts resolved, not the friction they felt. Next time, I'd put prototypes in front of operators sooner and track manual override rates as the primary metric for system trust.
04Enforce privacy at the architecture level
I'd strictly limit the LLM to pre-computed signals, ensuring no sensitive customer details ever hit an external model.