How a Fintech Team Cut AI Incidents by 94% in 6 Weeks

Introduction

In January of this year, a mid-size fintech company — a lending platform processing roughly 4,000 AI-assisted decisions a day — came to us with a problem. Their model was performing well by every benchmark they tracked. Accuracy on held-out test sets looked fine. BLEU scores, coherence ratings, user satisfaction surveys: all green. And yet every two or three days, something made it into production that shouldn't have. An incorrect loan term surfaced in a customer summary. A compliance disclosure got silently dropped. An edge-case user profile triggered an output that flagged in a manual audit.

The incidents weren't catastrophic. But they were consistent. And they were getting harder to explain to regulators who were starting to ask pointed questions about oversight.

We spent the first week not building anything. We spent it reading logs — specifically, the logs they didn't have.

The real problem: no observable failure surface

The team had a moderation call at the end of their pipeline. It returned a binary pass/fail. When it passed — which it did 99.4% of the time — the output went to the user and the system wrote a success event to the database. There was no intermediate signal. No record of what the model had actually been asked, what it returned before post-processing, or whether the output had been modified on its way out.

When an incident occurred, the investigation started from a user report and worked backwards through a system that had been designed to forget. It took an average of eleven hours to reconstruct what had happened. By that point, the same pattern had usually repeated dozens of times.

"The model wasn't broken. The observability was. You cannot fix what you cannot see, and they were flying blind at 4,000 completions a day."

What we built in six weeks

We did not replace their model. We did not swap their moderation provider. We built three things around the pipeline they already had.

Structured completion logs. Every call to the model now emits a log entry with the full input hash, output hash, token counts, latency, moderation verdict, and a normalized output category. These write to a queryable store, not a flat file. The mean investigation time dropped from eleven hours to twenty-two minutes within the first week — before a single guardrail change had been made.

Output schema validation. Their AI responses fed into downstream systems that expected specific fields. Occasionally the model would restructure its response in a way that omitted or renamed a field. The downstream system would either silently drop it or fill it with a default. Neither case was logged as an error. We added a Zod schema at the pipeline boundary. In the first 48 hours it caught 37 outputs that would have caused silent data corruption.

A behavioral regression suite. The team had no automated tests for model behavior — only for the application code around it. We wrote 60 prompt–expected-output pairs covering their highest-risk categories: compliance disclosures, loan term presentation, and out-of-scope user queries. These run on every deployment and on a nightly schedule against the live model. Two of the six weeks were spent writing these tests. They have caught three model-drift events since go-live.

What 94% actually means

Before: roughly 2.3 incidents per week reaching user-facing surfaces. After: an average of 0.14 per week over the 8 weeks since launch — and the ones that do surface are caught within minutes, not days. The compliance team now has a query they run every morning instead of a weekly manual audit. The engineering team ships model changes with the same confidence they ship application code.

None of the individual pieces were novel. Structured logging, schema validation, and behavioral tests are standard practice in every other part of software engineering. The gap was that nobody had applied them to the AI layer.