Banking LLM Architecture Inside a Regulated Perimeter

The constraint

Financial services does not ask "can the model do this." It asks: where does the data physically live, who can prove what the system decided in March, and what happens when the model is confidently wrong.

Those three questions reshape the architecture far more than any model choice.

Approach

The governing principle: the LLM is an untrusted component inside a trusted system. Treat it like a third-party API you do not control — because effectively, you don't.

That produces three hard boundaries:

1. A redaction boundary before the model. Cloud DLP inspects and tokenises every payload before it reaches inference. The model never receives a name, account number or identifier — it receives placeholders, and rehydration happens after the response returns, inside the trusted zone.
2. A deterministic rules layer in front. Anything with a definite answer — eligibility, limits, entitlements — is computed in code, not generated. The model handles language, never arithmetic or policy.
3. A validation boundary after. Responses are schema-checked and range-checked. A response that fails validation is discarded and the request falls back to a templated answer rather than degrading gracefully into a plausible-sounding fabrication.

Architecture

Everything sits inside a VPC Service Controls perimeter, so even a leaked credential cannot exfiltrate data to a public endpoint. Vertex AI is accessed through Private Service Connect, which keeps inference traffic off the public internet and satisfies residency requirements by pinning the region.

IAM is entirely policy-as-code in Terraform. This is not a purity argument — it is the only practical way to answer "who had access to this dataset last quarter" during an audit. Git history becomes the access log.

Every inference writes an immutable record: input hash, retrieved context IDs, model version, output, validation result. Not the raw content — the hash. You can prove what happened without creating a second copy of sensitive data.

What broke

The first version logged full prompts for debuggability. That is a compliance incident waiting to happen: you have just created an unmonitored secondary store of customer data with different retention rules from the primary system.

Replaced with content hashes plus context IDs. You lose the ability to eyeball a prompt in production, which genuinely hurts during an incident — so a separate, short-retention debug mode exists behind a break-glass IAM role that triggers an alert when used.

Outcome

Passed internal security review on the second pass. Zero PII in model prompts, verified by DLP scanning the outbound payloads independently. The audit trail answered a regulator question retrospectively without engineering involvement, which is the real measure of whether observability was designed rather than bolted on.