Making an AI feature audit-ready for a brokerage

01

Context

The brokerage's customer service team handles tens of thousands of secure messages a month about transfers, account features and statements. The firm had built an assistant that read each incoming message and drafted a reply for an agent to review. It worked well, and agents liked it, but it had been built quickly by a product team.

When internal audit added it to the annual plan, and a FINRA examination request asked about the firm's use of AI in customer communications, the firm could not answer basic questions: which model version wrote a given draft, what instructions it had been given, whether an agent changed the draft before sending, and how the firm knew a prompt change had not made answers worse.

02

Constraints

Regulatory

Customer communications had to be supervised and retained under FINRA's communications and books-and-records rules. The firm's model risk and change-management policies applied to the assistant like any other system, and internal audit needed evidence it could test independently.

Data

Messages contain customer personal and account information. Nothing could be sent to a model outside the firm's AWS environment, records had to be tamper-resistant, and test data used outside production had to be masked.

Integration

The controls had to fit around the existing assistant and the firm's case-management system without slowing agents down or changing their workflow.

Timeline

The evidence had to be ready for internal audit's fieldwork, ten weeks after we started.

03

What we built

We began by mapping how a reply was produced, from the incoming message to the sent response, and listing every point where something could change the outcome: the prompt template, retrieved help-center articles, the model and its settings, and the agent's edits.

We then made every draft fully traceable. Each one is now recorded with the exact prompt, the retrieved sources, the model identifier and version, the response, the agent who reviewed it, the changes they made and the time it was sent. Records go to write-once storage with retention set to the firm's communications policy, and can be searched by customer, agent or date.

Prompt templates and model settings moved into version control. A change now needs a pull request, a second reviewer, and a passing run of the evaluation suite we built: several hundred masked real-world messages with expected answers and checks for accuracy, required disclosures, prohibited statements and tone. A change that lowers the scores cannot be deployed.

We strengthened the human-in-the-loop controls so that drafts cannot be sent without an agent's action, high-risk topics (complaints, potential fraud, requests for advice) are flagged and routed to senior agents, and supervisors can sample sent replies for review from a queue.

Finally, we wrote the evidence pack with compliance and internal audit: a description of the system and its data flows, the control objectives and how each is met, the evaluation results over time, the incident procedure for a bad answer, and queries auditors can run themselves against the records.

• End-to-end records of every draft: prompt, sources, model version, response, agent edits and send time
• Write-once retention for AI records aligned with the firm's communications policy
• Prompt templates and model settings in version control with reviewed changes
• Evaluation suite of masked real messages, run in the pipeline to gate every change
• High-risk topic detection with routing to senior agents
• Supervisor sampling queue for sent replies
• Incident procedure and runbook for incorrect or harmful answers
• Evidence pack mapped to the firm's control framework, with self-service audit queries

04

Architecture

Expand

Reply path, human review, records and the change path for prompts and models. Firm details have been removed.

Full size✕

05

Results

Internal audit completed its review of the assistant with no findings, and the firm answered the FINRA examination request with records and evaluation history rather than descriptions. The evaluation suite has since caught two prompt changes that would have dropped a required disclosure, both blocked before release. The firm now uses the same pattern for every new AI feature.