# Validating the Unseen: A Framework for Managing Generative AI Risk in Financial Services
## Introduction
Imagine you are a risk analyst at a major financial institution, and a new AI tool has been proposed for your credit underwriting team. The system reads a borrower’s financial statements, pulls in relevant external research, and drafts the initial version of a credit assessment memo. On paper, it promises to save your analysts dozens of hours every week. The business unit is eager to deploy it within the next quarter.
You pull up the standard validation template — the one that has guided model reviews for over a decade, the one that has been stress-tested across scorecards, loss forecasting engines, and pricing algorithms. You reach the first required field:
*”Submit the development dataset.”*
There is no development dataset. The model was trained on a corpus no one at your firm has ever seen, built by a vendor who is unwilling or unable to describe its contents in detail. And this is just the opening question from a template containing ninety more.
This is the precise moment when model risk management, as a discipline, confronts a technology it was never built to handle. Generative AI has arrived in financial institutions faster than any previous modelling innovation, and it arrives not as a single, inspectable component but as an opaque system whose outputs vary with every interaction, whose core training data remains invisible, and whose failures are fluent, confident, and difficult to detect.
The craft of validation must therefore shift. It moves from the ability to replicate a model’s logic to the ability to design tests that expose its weaknesses. What follows is a practical framework for doing exactly that — one built on risk tiering, outcome-based evaluation, robustness testing, and continuous monitoring for silent drift. Though the context is financial services, the principles transfer directly to any high-stakes domain where an incorrect answer carries consequences that outweigh the cost of a delayed response.
—
## The Architecture of Model Risk Management in Finance
Before examining the unique challenges posed by generative systems, it helps to understand the established discipline they are being folded into.
Banks and other regulated financial institutions depend heavily on predictive models. These models determine who qualifies for credit and at what price. They set capital reserves, project losses under recessionary scenarios, value hard-to-liquidate positions, flag potentially fraudulent activity, and feed directly into published financial statements. When a traditional statistical model goes wrong, the fallout is not measured in frustrated customers — it is measured in mispriced risk, understated reserves, regulatory sanctions, and occasionally losses measured in the billions.
The industry learned these lessons at great expense. Credit models that assumed housing prices could not decline nationally were a material contributor to the 2008 financial crisis. A revised risk model at a single institution in 2012 understated its exposure so severely that a trading loss spiraled into the billions before anyone identified the root cause. In response, regulators formalized the discipline. Guidance issued in 2011 in the United States — commonly referenced as SR 11-7 — defined model risk as the potential for adverse consequences arising from decisions based on incorrect or misused model outputs, and mandated that financial institutions manage it deliberately and continuously.
In the European Union, the AI Act takes a parallel approach, codifying expectations for high-risk AI systems used in creditworthiness assessments, pricing decisions, and essential banking services. Its core obligations around risk governance, data documentation, transparency, human oversight, and accuracy align closely with the conceptual rigor demanded by earlier guidance. For global institutions operating across jurisdictions, a single validation framework can satisfy both regimes, though the AI Act introduces additional requirements around fundamental rights impact assessments and post-deployment monitoring that extend the traditional scope of oversight.
### The Three Lines of Defense
The model risk management structure at large financial institutions follows a consistent, well-established pattern:
– **First line**: The business units and model development teams. They build the models, test them internally, and own both their performance and their operational use.
– **Second line**: The model risk management and validation function. They independently challenge models before they receive approval, and they continue to challenge them throughout the model lifecycle.
– **Third line**: Internal audit. They verify that the first and second lines are executing their responsibilities effectively.
This article focuses on the second line — the independent validation function. A validator does not merely verify arithmetic. They ask whether the modelling approach was conceptually defensible, whether the data supported it, whether the output performs reliably when tested, whether the production implementation matches what was originally approved, and whether the people consuming the output understand its limitations. Nothing is permitted to go live without their formal sign-off, and every model undergoes periodic re-examination.
Three principles have anchored this review for over a decade and remain stable regardless of technological change:
1. **Conceptual soundness** — Is the underlying approach defensible given the problem it is solving?
2. **Outcomes analysis** — Does the output hold up when tested against independent data and expert judgment?
3. **Ongoing monitoring** — Is the model still performing as expected in production, and do the controls designed to catch degradation remain functional?
—
## Why Generative AI Breaks the Conventional Validation Playbook
Generative AI has entered banks at a pace unmatched by any previous modelling technology, and it has arrived in countless forms — complaint summarisation, policy lookup, research retrieval, first drafts of credit memos, internal documentation, and virtually anything else a knowledge worker might need. These tools are attractive because they directly influence operational cost, but they are also potentially dangerous, particularly because they are deployed in close proximity to customers and to credit decisions — precisely the contexts where a regulated institution has the least tolerance for an incorrect answer.
The validation apparatus that was designed to prevent this risk no longer fits. Every question on the standard template assumes properties that these systems do not possess.
### Five Structural Breaks
The incompatibility between traditional validation and generative AI can be distilled into five structural breaks that fundamentally change what it means to assess a model:
| Property | Why Traditional Validation Cannot Absorb It |
|—|—|
| **There is no single model — there is a system** | The output depends on query handling, retrieval mechanisms, prompt templates, the base model itself, decoding settings, guardrails, and orchestration logic. Changing any one of these components alters behaviour in ways that are difficult to isolate. |
| **The output is a distribution, not a point estimate** | Non-determinism is an intentional design feature. Even at the lowest temperature setting, batching effects and context variation produce differences in output. Two identical prompts can yield materially different results. |
| **There is no ground truth for open-ended tasks** | Traditional metrics like AUC, KS, Gini, and MSE all require an observable target variable. Summarisation, drafting, and research retrieval have no single correct answer against which to measure them. |
| **You did not build the core component** | The institution has no access to training data, no methodology document, and no ability to replicate the model. The vendor provides a system card and benchmark scores on tasks that may bear little resemblance to the intended use case. |
| **Version drift occurs without a formal change request** | Hosted models are updated behind stable API endpoints. Nothing in the institution’s change management process fires, because procedurally nothing has changed — and yet the behaviour may have shifted silently. |
The first break carries a direct procedural consequence: **the unit of validation is the system, not the model.** Two different use cases built on the same underlying model constitute two separate validations, each requiring its own evidence package.
The second break inverts an instinct that many practitioners carry from traditional modelling. A system that answers correctly ninety-five percent of the time but produces a confidently wrong answer the remaining five percent is a fundamentally different object from a model with a five percent error rate drawn from a known error distribution. Fluency is uncorrelated with correctness in a way that no calibrated probability-of-default model has ever been. A fluent, well-structured response that contains fabricated citations or incorrect figures is, in a regulated context, more dangerous than a clearly approximate one.
Because blanket validation is impossible in the face of these structural breaks, risk tiering becomes the primary lever that determines how much evidence is sufficient before deployment.
—
## Risk Tiering for Generative AI: A Practical Framework
Everything that follows — the evidence required, the depth of testing, the frequency of monitoring — scales from the risk tier assigned to a given use case. The classical drivers of model risk still apply: business reliance on the output, the severity of an error, the complexity of the system, and the feasibility of implementing controls. Two additional considerations matter specifically for generative AI:
### How far does the output travel?
The exposure level of a given deployment dramatically affects the risk profile:
– **Input to an internal decision**: When the system summarises research that feeds into a credit officer’s judgment, the human review step serves as the primary control. The risk is moderate because a trained professional remains in the loop.
– **Reaches a customer or regulator**: When the system generates correspondence, disclosures, or complaint responses, reputational and conduct risk become the dominant concern. The output is no longer contained within the institution — it enters the public or regulatory domain.
### Does it produce text, or does it also act?
A system that drafts content for a human to read carries a fundamentally different risk profile from a system that calls external tools, writes to systems of record, or triggers downstream automated processes. The latter introduces the possibility of irreversible actions taken on the basis of unreliable output.
—
## Three Questions an AI Risk Assessment Report Must Answer
The questions a validator asks have not changed. What has changed is the nature of the evidence required to answer them. The assessment must address three fundamental questions:
1. **Is the system appropriate for its intended purpose?** This is not about whether the underlying model is “good” in the abstract — it is about whether the specific configuration, retrieval setup, guardrails, and human oversight structure are fit for the task at hand.
2. **Does the system perform reliably under realistic conditions?** This requires moving beyond aggregate accuracy scores and examining where, how, and why the system fails.
3. **Can we detect and respond to degradation over time?** This demands a monitoring architecture capable of catching silent drift — changes in behaviour that occur without any formal update to the system.
—
## Assessing What We Can and Cannot Know
### Should This Even Be a Language Model?
The cheapest and most valuable challenge available to any validation team is to ask whether a language model is the right tool for the job. A surprising number of proposed use cases are fundamentally deterministic problems — parsing structured data, extracting specific fields, classifying inputs into known categories — where a generative model introduces unnecessary variance to solve something a rule-based parser handles with exactness. If a simpler method delivers ninety percent of the value with none of the fabrication risk, that comparison belongs prominently in the assessment report. Complexity must earn its place.
### What Exactly Is the System?
The validation team needs a complete component map of the system they are reviewing, along with a clear picture of which components the institution actually controls versus those managed by the vendor.
| Component | Who Controls It | What Can Go Wrong |
|—|—|—|
| Retrieval and indexing | The institution | Stale or missing evidence; overly broad access to internal documents |
| Prompt template | The institution | Ambiguous instructions; exposure to injection attacks |
| Base model | The vendor | Everything — this is the opaque core |
| Decoding settings | The institution | Output dispersion controlled by temperature and sampling choices |
| Guardrails | The institution | Both over-blocking (rejecting valid outputs) and under-blocking (letting harmful outputs through) |
| Orchestration logic | The institution | Compounding errors across sequential steps |
In practice, retrieval quality and prompt construction tend to dominate system performance in production. The base model is often the least significant source of variance — which is fortunate, because it is the component that cannot be inspected.
### What Went Into It?
The specification cannot simply be “we used a foundation model.” Every customisation choice requires explicit justification:
– **Prompting versus fine-tuning**: Why was this route selected, what is the ongoing maintenance burden, and what is the cost of getting it wrong?
– **Prompt content and structure**: Prompts in version control are model logic, and they must be treated as such.
– **Decoding settings**: Temperature and sampling parameters directly control output dispersion. Changing them constitutes a model change and should trigger a re-review.
– **Context window limits**: Does the real-world input fit within the model’s context limit, and what happens when it does not?
– **Tuning data**: Where did any fine-tuning data originate, does it contain confidential information, and how was quality verified?
### What Can We Honestly Explain?
Traditional model explanation tools provide limited utility for generative systems. In practice, two approaches have proven workable:
The first is **claim-level attribution**, where each factual assertion in the model’s output is checked against a specific passage in the source material. This is achievable, auditable, and useful in a regulated workflow because it provides a verifiable trail.
The second — fully characterising how the system behaves across every possible input — is not achievable. The practical substitute is behavioural testing over a structured set of cases. You deliberately expose the system to diverse, controlled inputs, observe how it responds, and generalise carefully from those observations. You are profiling the system, not proving its correctness.
—
## Outcome Analysis: Evaluating Quality for Language Models
The outcome analysis is the most critical phase of the entire validation process. This is where the team tests whether the system delivers what it promises and, just as importantly, where it fails.
### Metrics Must Match the Task
The evaluation dimensions should be selected based on the specific use case, not applied generically:
– **Summarisation**: Assess completeness of coverage, factual consistency with the source material, and fluency of expression.
– **Retrieval-augmented answering**: Evaluate whether claims are supported by retrieved context, whether the answer adheres to the query instructions, and whether the retrieved material was both sufficient and free of noise.
– **Open-ended generation**: Score instruction following, register and tone appropriateness, verbosity, and factual accuracy against external references.
These high-level dimensions are a starting point. A credible validation report must score individual outputs on explicit, separately reported dimensions. Collapsing quality into a single aggregate number obscures the specific ways a system can fail, and in a regulated environment, that opacity is itself a risk.
### The Full Dimension Set
The evaluation dimensions naturally divide into two groups.
**Group one: Is the output factually reliable?**
– **Factual truthfulness**: Are claims about the subject matter correct? Scored against an expert-labelled reference set, claim by claim.
– **Hallucination rate**: Does the system invent entities, figures, citations, or legal clauses that do not exist? Fabricated citations can be detected automatically, but other forms of fabrication require human review.
– **Groundedness**: Does every claim trace back to supplied evidence? This is scored using claim-to-source entailment analysis.
– **Completeness**: Was any required element of the query omitted? Assessed against a rubric checklist.
– **Relevance**: Does the output answer the question that was actually asked? Also assessed via rubric.
These three dimensions — truthfulness, hallucination, and groundedness — are frequently collapsed into a single number, but they should not be. An answer can be perfectly grounded in an outdated document: technically grounded but factually wrong in the current context. An answer can be factually true but ungrounded, which renders it unusable in a regulated workflow because no reviewer can verify its provenance. In credit and compliance contexts, groundedness is typically the binding constraint, because verifiability is the control mechanism.
**Group two: Is the output usable and safe in a regulated workflow?**
– **Instruction following**: Were all explicit constraints in the prompt honoured — format, length, required sections, excluded content?
– **Format compliance**: Does the output parse correctly into whatever structure the downstream process expects?
– **Verbosity**: Is the response length proportionate to the task, or does it contain unnecessary padding?
– **Writing quality**: Is the output clear, well-structured, and appropriate in tone for the audience?
– **Refusal appropriateness**: Does the system correctly decline requests it should refuse while answering those it should address? Tested against two separate case sets.
– **Bias in output**: Does output quality shift depending on protected attributes present in the input? Measured through matched-pair comparisons.
– **Stability**: Does the same input produce materially the same answer across repeated draws? Measured using semantic similarity scoring.
### Robustness Testing
Robustness is assessed by perturbing inputs in ways that preserve their meaning and verifying that the output does not change materially. Effective perturbation strategies include:
– **Source text modifications**: Synonym substitution, introduced typos, reformatting of the evidence document.
– **Query phrasing variations**: Asking the same question three different ways.
– **Evidence supply changes**: Reordering passages, injecting irrelevant material, or removing key sources.
Reordering the evidence is among the highest-value tests on this list. If the answer changes when nothing about the evidence has changed, a genuine structural weakness has been identified.
### Where It Fails, Not Just How Often
Aggregate scores obscure localised failures. A more informative approach embeds the input population into meaningful clusters and scores performance per cluster. Weak segments typically reveal themselves as semantically coherent groups — a specific document type, a particular product category, a distinctive phrasing style. Once identified, these segments can be scoped out of the system’s operating domain or specifically addressed through targeted fixes.
### Hallucination Detection at Scale
Three proven methods exist for detecting fabrication at scale:
1. **Cross-reference verification**: Checking generated claims against a known source corpus to identify unsupported assertions.
2. **Self-consistency checks**: Generating multiple responses to the same query and flagging inconsistencies.
3. **Verification questioning**: Breaking complex claims into sub-questions and verifying each one independently.
### Validating the Evaluator
Most organisations use a second language model to score outputs against a quality rubric. That judge is itself a model producing an output that informs a control decision — and it must be validated with the same rigor applied to the primary system.
Key judge tests include:
– **Agreement with human raters**: Establishing that the automated scorer measures what it intends to measure, with a quantifiable error bound.
– **Position and ordering bias**: Verifying that the judge does not systematically favour whichever option appears first or last.
– **Verbosity bias**: Confirming that longer answers do not receive higher scores regardless of quality.
– **Self-preference**: Checking whether the judge favours outputs from its own model family.
– **Periodic recalibration**: Recognising that the judge itself drifts over time and must be recalibrated on a defined schedule.
Deploying automated scoring without first validating the scorer does not reduce model risk — it relocates it.
—
## Safeguards for Generative AI in Production
It is not possible to prevent a foundation model from producing fabricated content entirely. What can be done — and what must be done — is to build a system that catches or bounds fabrication before it reaches an end user or influences a consequential decision.
The safeguards operate across multiple layers, each addressing a different attack surface or failure mode:
| Layer | Function | Typical Failure Without It |
|—|—|—|
| **Access controls** | Restricts who can invoke the system | Unqualified users treating output as authoritative |
| **Scope limitation** | Locks functionality to the approved use case | A general-purpose model quietly being used for unintended tasks |
| **Input screening** | Shapes and constrains what enters the system | An unbounded query space exposing the system to injection attacks |
| **Generation constraints** | Controls how the model responds — settings, length, required structure | Lengthy, unfocused output that no one reads properly |
| **Output screening** | Reviews output before it reaches any user or system | Errors reaching customers or regulators |
| **Human review** | A certified reviewer positioned between the output and the decision | A control that exists on paper only |
| **Logging and recording** | Captures every interaction for monitoring and investigation | No ability to detect drift or misuse after the fact |
None of these layers alone is sufficient. The defence-in-depth approach ensures that a failure at any single point does not cascade into a consequential error.
—
## Ongoing Monitoring for Silent Drift
A monitoring plan must be defensible — meaning it should specify a testing frequency, define indicators mapped to actual risks, establish thresholds that trigger escalation, provide rules for when indicators conflict, and outline a pre-agreed action plan when thresholds are breached.
Key monitoring indicators include:
– **Fabrication and tone rates on sampled output**: The most direct signal of behavioural drift, including changes caused by silent upstream version updates.
– **Query pattern stability**: Detects whether users are gradually drifting outside the approved use case into unvalidated territory.
– **Retrieval quality**: Flags index staleness or corpus changes that degrade the grounding of responses.
– **Generation success rate**: Tracks attempted versus completed generations as a measure of operational health.
– **User feedback**: Often the cheapest, most continuous, and earliest signal available — if the feedback mechanism is in place and functioning.
– **Override and edit rate**: Measures whether the human review control is actually functioning, not merely whether it exists. A steadily declining override rate over six months is not evidence of model improvement — it is often evidence that the review control has become performative.
Monitoring is not a one-time exercise. It is an ongoing discipline that must be embedded in the operational rhythm of any organisation deploying generative AI in a production environment.
—
## Frequently Asked Questions
### 1. Can generative AI models be validated the same way as traditional statistical models?
No. Traditional model validation assumes access to training data, a replicable methodology, and a deterministic relationship between inputs and outputs. Generative AI systems violate all three of these assumptions. Validation must shift from replication to test design — constructing scenarios that probe the system’s behaviour rather than attempting to rebuild it.
### 2. What is the single most important thing a validator should ask about a generative AI system?
Whether the system’s core component — typically the base model — is one the institution can inspect, audit, or replicate. If the answer is no, the entire validation strategy must compensate by testing more rigorously, monitoring more continuously, and designing guardrails that assume the model can produce incorrect or fabricated output at any time.
### 3. How do you measure the quality of a system that has no ground truth?
You measure it on multiple dimensions — factual truthfulness, grounding, completeness, instruction following, format compliance — and you evaluate each dimension separately. You also use behavioural testing: deliberately exposing the system to controlled variations in input and observing whether the output changes in ways that indicate instability or unreliability.
### 4. What is “silent drift” and why is it dangerous?
Silent drift refers to changes in a model’s behaviour that occur without any formal update to the system — typically because the underlying hosted model was updated by the vendor behind a stable API endpoint. It is dangerous because it bypasses the institution’s change management process, meaning the validation performed at deployment may no longer reflect the system’s current behaviour.
### 5. Should every proposed use of generative AI in a bank be approved?
Not necessarily. A significant fraction of proposed use cases are deterministic problems where a generative model introduces unnecessary variance. The first challenge should be whether a simpler, more reliable method exists. If it does, that comparison should be documented and the use case may not require the full generative AI validation apparatus.
### 6. How does the EU AI Act change the validation process compared to earlier guidance?
The AI Act introduces explicit requirements for fundamental rights impact assessments and post-market monitoring that go beyond the traditional scope of model risk management. For global banks, the underlying principles remain consistent, but the documentation and monitoring obligations are more prescriptive, particularly around transparency and the rights of individuals affected by automated decisions.
### 7. What is the role of the human reviewer in a generative AI workflow?
The human reviewer is the final control — the last line of defence between the model’s output and a consequential decision. Their role must be defined clearly: what they are checking for, how they are trained, what they are authorised to approve or reject, and what happens when they disagree with the system’s output. A human review step that exists only on paper provides no real control.
### 8. How often should monitoring thresholds be reviewed?
Monitoring thresholds should be reviewed at least annually, and immediately whenever there is a material change to the system’s architecture, the underlying model, the retrieval corpus, or the use case profile. Thresholds set at deployment may become either too sensitive or too permissive as the system’s operating environment evolves.
—
## Conclusion: The Evolving Discipline of Model Risk Management
Two foundational principles anchor everything in this framework, and both are more relevant now than ever before.
**No model is absolutely valid.** Validation demonstrates that a model is suitable for a specific purpose under defined conditions — it does not prove the model is correct in all circumstances. For generative systems, this point carries extra weight because the otherwise unlimited output space is constrained only by the purpose and the conditions under which the system was tested. Anything outside that tested envelope remains unvalidated.
**More complexity is not always better — and this applies equally to the governance built around these systems.** A low-materiality AI assistant does not require a hundred-page validation report. The rigour of the validation should be proportional to the risk the system presents, not to the novelty of the technology.
Generative AI does not render model risk management obsolete. It makes it more essential than ever. The second line’s role evolves from model replication to test design, from single-model evaluation to system-level assurance. The fundamental questions have not changed. Only the evidence required to answer them has.
—
Thank you for reading
