Your company rolled out Claude six months ago. Most of the org is on the Team plan, signed in over SSO. Two product teams hit the API directly from CI jobs. Last quarter, legal asked a simple question: which of our customer support agents have pasted ticket bodies into a chat to draft replies?
Nobody could answer it. Not the security team, not IT, not the engineers who set up the subscriptions. The data went out. What it contained is on no one's side. There is no log to pull, no query to run, no audit trail to hand to counsel.
That gap is the default state of every enterprise running Claude today, and not because the tooling to close it is missing. The proxy layer that does the job is open source, runs in a container, and almost nobody puts it in front of their LLM traffic.
The unmanaged AI surface area
Most companies adopting Claude end up with three independent egress paths to the same vendor. Backend services hit the Anthropic API directly from application code and CI. Most employees use the web app, signed into the company's Team or Enterprise plan over SSO. Developers run Claude Code and third-party IDE plugins that call the API with workspace keys minted from the same admin console.
Each path is an exit door for whatever a human or a process decides to paste. Source code with hardcoded API keys. Customer PII in a ticket excerpt. A board deck pasted in for a summary. Internal pricing strategy dropped into a "rewrite this" prompt. The model does not need any of that to do its job. It receives it anyway.
Your existing controls do not cover this surface. Whatever the security team runs at the network layer sees an outbound HTTPS request to Anthropic and nothing inside it. SSO gets users signed in to the Team plan but does not inspect what they type once they are there. Anthropic's admin console shows you seat-level usage and spend, not the contents of any individual prompt. Nothing inside your network can answer "what went out."
The result is a vendor relationship that already produces real business value and zero evidence that you used it safely. If your compliance program leans on auditability, that gap is a problem you cannot defer.
What a proxy gateway changes
A proxy gateway collapses the surface back to one ingress point. Every client, every team, every CI job, every Claude Code session, every internal tool, all of it makes its requests to one URL inside your network. From there the proxy makes the outbound call to Anthropic.
The architecture is unremarkable. The consequences are not.
One ingress means one place to put policy. PII detection runs once, against every prompt, regardless of whether the caller is a human in a chat UI or a backend service in a deploy pipeline. Secret detection runs the same way. Per-team rate limits and per-department spend caps apply uniformly across the API, the web app, and Claude Code, instead of living in three separate admin consoles that nobody reconciles.
One ingress also means one log. Every request and every response (subject to your retention policy) lands in a structured record you can grep, attribute to a user, and hand to legal when they ask the question I opened with. The audit trail is the part that flips AI adoption from "we hope it's fine" to "we can defend our usage."
LiteLLM is the open-source implementation of this pattern that most teams I respect have converged on. It speaks the OpenAI API format on the inbound side, routes to Claude, Bedrock, Azure, Gemini, and 100 other providers on the outbound side, and ships the policy primitives the audit story needs.
LiteLLM in practice
The piece that earns its keep is the guardrails system. The open-source content filter ships with prebuilt regex patterns for the obvious sensitive tokens (SSN, email, phone, credit card, AWS access keys) and lets you wire each one to either BLOCK the request outright or MASK the sensitive value before the prompt reaches the model. Filters can run pre-call, mid-stream, post-call, or all three.
A minimal config looks like this:
guardrails:
- guardrail_name: 'pii-pre-call'
litellm_params:
guardrail: litellm_content_filter
mode: 'pre_call'
patterns:
- pattern_type: 'prebuilt'
pattern_name: 'us_ssn'
action: 'BLOCK'
- pattern_type: 'prebuilt'
pattern_name: 'email'
action: 'MASK'
- pattern_type: 'prebuilt'
pattern_name: 'aws_access_key'
action: 'BLOCK'
blocked_words:
- keyword: 'confidential'
action: 'BLOCK'
description: 'Sensitive internal information'Here is what that does to a real prompt:
# Original prompt from a support agent
"Draft a reply to [email protected] confirming her SSN 123-45-6789 is on file."
# pre_call fires before anything leaves your network:
# us_ssn -> BLOCK -> proxy returns 400 to the caller
# The model never sees the request, and the log records who tried.
# If only the email MASK rule applied (no SSN in the prompt), the model sees:
"Draft a reply to <EMAIL_ADDRESS> confirming her record is on file."The full pattern list and YAML reference live in the guardrails quick start.
Regex catches the structured stuff. For unstructured PII (names, addresses, free-form medical notes, the things humans paste without thinking) you bolt on a real classifier. LiteLLM ships first-class integrations for Microsoft Presidio, AWS Bedrock Guardrails, Azure Content Safety, Lakera, Pillar Security, and a handful of others. Same config shape, same policy slot, different detection engine underneath. Bedrock Guardrails also covers denied topics and prompt injection, which the regex filter cannot.
Above the guardrail layer you get virtual API keys. Each user, team, or service gets their own key, scoped to a subset of models, a monthly spend cap, and a requests-per-minute limit. The proxy attributes every log line to that key, so "who pasted what" becomes a query instead of a guess. JWT-based mapping ties the key back to a real SSO identity, which is how you make the audit story stand up when legal actually shows up.
When a Bedrock region goes down, the same config does fallback routing, so traffic shifts to Claude on Anthropic's API without a code change in the calling service. Same surface. Different upstream.
Honest tradeoffs
LiteLLM is not a free win, and the post would not be useful if I pretended otherwise.
The open-source core covers the content filter, virtual keys, logging, and provider routing. Multi-tenant policy attachment (different guardrails per team, conditional policies by model or header, policy inheritance) sits behind the Enterprise tier. For a small org with one platform team and uniform policy, the open-source feature set is enough. For a large org with finance, healthcare, and engineering each needing their own rule set, you will hit the Enterprise paywall faster than you expect. Budget for it.
The built-in content filter is regex-only. Regex is fine for structured tokens like SSNs and credit cards. It cannot find a freeform sentence containing a customer name and a diagnosis, because the pattern does not exist to match against. Production-grade PII coverage on unstructured prose means pairing the content filter with Presidio or Bedrock Guardrails. That is another service to operate and another hop of latency on every call. Budget for both.
The proxy is another piece of infrastructure you operate. Docker container, Kubernetes deployment, monitoring, patching, on-call rotation. The win is real, but the operational cost is not zero, and "we already have too many services" is a legitimate counterargument that gets resolved by scoping the rollout, not by ignoring it.
The proxy catches leaks at the wire. It does not change the culture that produced the leak. Training, acceptable-use policy, and clear examples of what not to paste still matter. The proxy is the safety net under the trapeze, not the trapeze.
The decision is already on the table
The real question is whether you can defend the data you have already sent out without a proxy in front of Claude.
If the honest answer is no, the gap closes faster than the rollout suggests. A weekend stand-up gets you most of the way. The Enterprise tier exists for the rest. The cost of adding the layer is small and concrete. The cost of leaving it off is large and unmeasured, which is the shape of every risk that wins arguments after the incident and loses them before.
I would rather have the audit trail.