What PMs Get Wrong About AI Reliability

Introduction

Ask a product manager who owns AI reliability in their organization and most will point to engineering. Ask the engineers and they will point to the model provider. Ask the provider and they will point to the evaluation suite that nobody has written yet. Meanwhile, the product spec has shipped, the feature is in front of users, and the first incident is three weeks away.

This is not a technology problem. It is a responsibility problem that starts at the spec stage — and product managers are the ones who set the spec.

The decisions you make before engineering starts

Reliability in an AI feature is not primarily determined by which model you use, which guardrail library you install, or how many review cycles the prompt goes through. It is determined by four decisions that get made — usually implicitly — at the spec stage, before engineering writes a line of code.

What is the output surface? Free text, structured data, a binary recommendation, or a ranked list each carry fundamentally different reliability profiles. Free text is the hardest to validate and the easiest to get wrong. If your spec calls for free-text output from an AI in a high-stakes context, that is a product decision with reliability implications, not a detail to resolve later.

What does failure look like? Most specs describe the happy path. Few describe what the feature should do when the model returns something unexpected, out-of-scope, or structurally malformed. If the failure mode is not in the spec, it will not be in the implementation. It will be discovered in production.

What is the acceptable error rate? "The AI should be accurate" is not a requirement. It is a wish. A requirement looks like: the feature must flag outputs for human review when confidence is below a defined threshold, and the false-negative rate must not exceed 0.5% on the test distribution. If you have not set a number, engineering cannot build to one.

Who reviews edge cases before launch? In non-AI features, QA owns edge case coverage. In AI features, the edge case space is effectively infinite and requires domain expertise to define. If the spec does not name a person who is responsible for writing and reviewing adversarial test cases before launch, it does not get done.

"Most specs describe the happy path in careful detail and leave the failure modes to inference. The model fills in the gaps. Not always in the way you'd want."

What a reliability-aware spec looks like

A spec that takes reliability seriously does not need to be longer. It needs to contain a few things that most specs omit entirely.

1. Explicit failure modes — at least three scenarios where the model might return something unexpected, and a specified behaviour for each

2. A defined confidence threshold — at what point does the output go to a human instead of directly to the user?

3. An output type commitment — free text, structured JSON, binary, or ranked; with a rationale for the choice given the reliability requirements

4. A named reviewer for adversarial test cases — someone with domain knowledge who is not on the engineering team

5. A success metric that includes failure rate — not just engagement or task completion, but how often the feature produces outputs that should not have reached the user

Why this is a product decision, not an engineering one

Engineering can build whatever reliability architecture you specify. They can add schema validation, observability hooks, confidence thresholds, fallback paths, and behavioural test suites. What they cannot do is decide what the acceptable error rate is for your specific users in your specific context. They cannot decide whether a misclassification in your product is a minor annoyance or a compliance incident. They cannot decide who in your organization is accountable when the model is confidently wrong.

Those are product decisions. They require understanding the user, the business context, and the risk profile of failure — which is exactly what a product manager is supposed to bring to the table.